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Why  GAO  Did  This  Study 


What  GAO  Found 


The  Department  of  Defense  (DOD) 
expects  the  cost  to  develop  and 
procure  the  major  weapon  systems 
in  its  current  portfolio  to  total 
$1.6  trillion.  With  increased 
competition  for  funding  within 
DOD  and  across  the  federal 
government,  effectively  managing 
these  acquisitions  is  critical.  Yet 
DOD  programs  too  often 
experience  poor  outcomes — like 
increased  costs  and  delayed 
fielding  of  needed  capabilities  to 
the  warfighter. 

In  2006,  this  Committee  mandated 
that  GAO  report  on  DOD’s 
processes  for  identifying  needs  and 
allocating  resources  for  its  weapon 
system  programs.  In  2007,  GAO 
reported  that  DOD  consistently 
commits  to  more  programs  than  it 
can  support.  This  follow-on  report 
assesses  DOD’s  funding  approach, 
identifies  key  factors  that  influence 
the  effectiveness  of  this  approach, 
and  identifies  practices  that  could 
help  improve  DOD’s  approach. 


DOD  often  does  not  commit  full  funding  to  develop  its  major  weapon  systems 
when  they  are  initiated,  despite  the  department’s  policy  to  do  so.  For  a 
majority  of  the  weapon  system  programs  GAO  reviewed,  costs  have  exceeded 
the  funding  levels  initially  planned  for  and  reflected  in  the  Future  Years 
Defense  Program  (FYDP) — DOD’s  investment  strategy.  To  compensate  for 
these  shortfalls,  DOD  makes  unplanned  and  inefficient  funding  adjustments, 
like  moving  money  from  one  program  to  another,  deferring  costs  into  the 
future,  or  reducing  procurement  quantities. 


Funding  Shortfalls  at  the  Start  of  Development  for  Five  Major  Weapon  System  Programs 
Program 
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Level  of  funding  established  in  the  FYDP  in  the  year  the  program  was  initiated 
Level  of  funding  the  program  needed  to  be  fully  funded  in  the  initial  FYDP 
Funding  required  beyond  the  initial  FYDP  to  complete  development 
Source:  DOD  (data);  GAO  (analysis  and  presentation). 


To  conduct  its  work,  GAO  assessed 
20  major  weapon  programs  in 
DOD’s  current  portfolio — 5  in 
detail — and  reviewed  relevant  DOD 
policy  and  guidance,  prior  GAO 
work,  and  other  relevant  literature. 
GAO  also  reviewed  the  practices  of 
selected  successful  companies. 


What  GAO  Recommends 


GAO  is  making  three 
recommendations  aimed  at 
increasing  funding  stability  and 
improving  acquisition  outcomes. 
DOD  believes  that  current  policies 
and  initiatives  sufficiently  address 
the  first  two  recommendations,  and 
did  not  concur  with  the  third. 

To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  GAO-08-619. 

For  more  information,  contact  Michael  J. 
Sullivan  at  (202)  512-4841  or 
sullivanm@gao.gov. 


DOD’s  flawed  funding  process  is  largely  driven  by  decision  makers’ 
willingness  to  accept  unrealistic  cost  estimates  and  DOD’s  commitment  to 
more  programs  than  it  can  support.  DOD  often  underestimates  development 
costs — due  in  part  to  a  lack  of  knowledge  and  optimistic  assumptions  about 
requirements  and  critical  technologies.  At  the  same  time,  DOD’s  continued 
failure  to  balance  its  needs  with  available  resources  promotes  unhealthy 
competition  among  programs  for  funding.  This  creates  incentives  for  service 
and  program  officials  to  establish  requirements  that  make  their  particular 
weapon  systems  stand  out,  with  less  consideration  of  the  resources  needed  to 
develop  them.  Ultimately,  DOD  tends  to  push  the  need  for  funding  to  the 
future  rather  than  limit  program  length  or  adjust  requirements. 

The  successful  commercial  companies  that  GAO  has  previously  reviewed 
achieve  adequate  and  stable  funding  for  product  development  programs  by 
following  a  disciplined,  knowledge-based  approach  to  estimating  program 
costs;  using  manageable  development  cycles  to  increase  the  predictability  of 
funding  needs  and  the  likelihood  of  program  success;  and  using  portfolio 
management  practices  to  make  decisions  about  which  programs  to  pursue. 
Once  programs  are  approved,  these  companies  firmly  commit  to  fully  fund 
them. 

_ United  States  Government  Accountability  Office 


Contents 

Letter 

1 

Results  in  Brief 

3 

Background 

5 

Failure  to  Commit  Full  Funding  to  Weapon  Systems  Contributes  to 
Poor  Acquisition  Outcomes 

9 

Unrealistic  Cost  Estimates  and  a  Failure  to  Balance  Needs  with 
Available  Resources  Underlie  DOD’s  Flawed  Funding  Approach 

12 

Proven  Practices  Help  Ensure  Accurate  Cost  Estimates  and 

Adequate  Program  Funding 

22 

Conclusions 

29 

Recommendations  for  Executive  Action 

29 

Agency  Comments  and  Our  Evaluation 

30 

Appendix  I 

Objectives,  Scope,  and  Methodology 

34 

Appendix  II 

Comments  from  the  Department  of  Defense 

37 

Related  GAO  Products 

42 

Table 

Table  1:  Development  Cost  Estimates  and  Baselines  for  20  Major 
Weapon  System  Programs 

14 

Figures 


Figure 

Figure 

Figure 

Figure 

Figure 


1:  Simplified  View  of  PPBE  Process  6 

2:  Shortfalls  at  the  Start  of  Development  for  Five  Major 
Weapon  System  Programs  10 

3:  Costs  Remaining  versus  Annual  Appropriations  for  Major 
Defense  Acquisitions  20 

4:  Notional  Comparison  of  Cost  Estimating  Uncertainty  and 
Levels  of  Knowledge  at  Program  Start  24 

5:  Range  of  Possible  Costs  Narrows  as  Knowledge  Is  Gained  25 


Page  i 


GAO-08-619  Defense  Acquisitions 


Abbreviations 


AOA  analysis  of  alternatives 

CAIG  Cost  Analysis  Improvement  Group 

CARD  Cost  Analysis  Requirements  Description 

DAS  Defense  Acquisition  System 

DOD  Department  of  Defense 

EFV  Expeditionary  Fighting  Vehicle 

FCS  Future  Combat  System 

FYDP  Future  Years  Defense  Program 

IDA  Institute  for  Defense  Analysis 

JCIDS  Joint  Capabilities  Integration  and  Development  System 

JSF  Joint  Strike  Fighter 

JTRS  Joint  Tactical  Radio  System 

MDA  Milestone  Decision  Authority 

MMA  Multi-mission  Maritime  Aircraft 

OMB  Office  of  Management  and  Budget 

OSD  Office  of  the  Secretary  of  Defense 

PPBE  Planning,  Programming,  Budgeting,  and  Execution 

SAR  selected  acquisition  report 

SIBRS  Space-Based  Infrared  System-High 

WIN-T  Warfighter  Information  Network-Tactical 


This  is  a  work  of  the  U.S.  government  and  is  not  subject  to  copyright  protection  in  the 
United  States.  It  may  be  reproduced  and  distributed  in  its  entirety  without  further 
permission  from  GAO.  However,  because  this  work  may  contain  copyrighted  images  or 
other  material,  permission  from  the  copyright  holder  may  be  necessary  if  you  wish  to 
reproduce  this  material  separately. 


Page  ii 


GAO-08-619  Defense  Acquisitions 


Accountability  *  Integrity  *  Reliability 


United  States  Government  Accountability  Office 
Washington,  DC  20548 


July  2,  2008 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Department  of  Defense  (DOD)  expects  the  cost  to  develop  and 
procure  the  major  weapon  systems  in  its  current  portfolio  to  total 
$1.6  trillion,  $335  billion  of  which  is  expected  to  be  spent  over  the  next  5 
years.  Effective  management  of  the  costs  of  these  acquisitions  is  critical 
given  the  increased  competition  for  funds  within  the  department  to 
support  ongoing  military  operations  in  Afghanistan  and  Iraq,  as  well  as 
growing  pressures  to  reduce  overall  DOD  spending  due  to  the  long-term 
fiscal  imbalances  facing  the  federal  government.  However,  many  of  DOD’s 
major  weapon  system  development  programs  have  experienced  poor 
outcomes — cost  increases  that  add  up  to  hundreds  of  millions  of  dollars, 
schedule  delays  that  add  up  to  years,  and  capabilities  that  fall  short  of 
what  was  promised. 

How  DOD  manages  its  weapon  system  investments  has  been  a  matter  of 
congressional  concern  for  many  years.  In  fiscal  year  2006,  the  Senate 
Armed  Services  Committee  raised  concerns  about  DOD’s  poor  track 
record  with  acquisition  programs  and  directed  GAO  to  assess  how  DOD’s 
processes  and  practices  for  identifying  requirements  and  allocating 
resources  affect  the  department’s  weapon  system  acquisition  programs.  In 
March  2007,  we  reported  that  DOD  lacks  an  effective,  integrated  portfolio 
management  approach  that  takes  into  account  all  of  the  department’s 
major  weapon  system  programs  and  that  requires  tough  decisions 
commensurate  with  available  resources.1  In  short,  we  noted  that  DOD 
commits  to  more  programs  than  it  can  support.  This  report,  also  done  in 
response  to  the  same  Senate  mandate,  focuses  on  DOD’s  funding  process 
and  its  impact  on  major  acquisitions.  Specifically,  the  report  (1)  assesses 


1  GAO,  Best  Practices:  An  Integrated  Portfolio  Ma?iagement  Approach  to  Weapon  System 
Investments  Could  Improve  DOD’s  Acquisition  Outcomes,  GAO-07-388  (Washington,  D.C.: 
Mar.  30,  2007). 
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how  DOD  budgets  for  and  funds  its  major  weapons  system  acquisition 
programs,  (2)  identifies  key  factors  that  influence  the  effectiveness  of  this 
approach,  and  (3)  identifies  proven  processes  and  practices  that  could 
help  improve  DOD’s  ability  to  effectively  allocate  resources  to  its 
acquisition  programs.  GAO  also  has  ongoing  related  work  specifically 
assessing  DOD’s  process  for  identifying  and  prioritizing  warfighting 
capability  requirements. 

To  assess  DOD’s  funding  process  and  to  identify  key  factors  that  influence 
the  effectiveness  of  that  approach,  we  reviewed  relevant  DOD  policy 
guidance,  legislation,  and  academic  literature,  and  assessed  cost  estimates 
and  budget  data  for  20  of  the  95  major  weapons  programs  in  DOD’s 
current  portfolio — which  represent  more  than  one-third  of  the  total 
expected  cost  of  DOD’s  current  portfolio  of  major  weapon  system 
programs.  To  gain  further  insights  into  the  impact  of  DOD’s  funding 
process  on  individual  programs,  we  conducted  more  detailed  analysis  for 
five  of  these  programs:  Global  Hawk,  Joint  Strike  Fighter  (JSF),  Future 
Combat  System  (FCS),  Warfighter  Information  Network-Tactical  (WIN-T), 
and  Multi-mission  Maritime  Aircraft  (MMA).2  In  addition  we  interviewed 
numerous  officials  from  the  Office  of  the  Secretary  of  Defense  (OSD)  as 
well  as  military  service  cost  analysis,  budgeting,  and  acquisition  offices.  To 
identify  proven  cost  estimating  and  budgeting  processes  and  practices  that 
could  be  used  by  DOD  to  improve  its  resource  allocation  process,  we 
utilized  information  from  our  March  2007  best  practices  report,  and 
conducted  follow-up  interviews  with  officials  from  three  of  the  five 
companies  that  provided  input  to  that  report — Eli  Lilly,  IBM,  and 
Motorola.  We  also  relied  on  our  Cost  Assessment  Guide,  which  provides  a 
cost-estimating  methodology  based  on  best  practices.3  We  conducted  this 
performance  audit  from  June  2007  to  May  2008  in  accordance  with 
generally  accepted  government  auditing  standards.  Those  standards 
require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate 


3  Global  Hawk,  an  Air  Force  unmanned  aircraft  system,  is  intended  to  provide  intelligence, 
surveillance,  and  reconnaissance  capabilities.  JSF  is  a  joint  Air  Force,  Navy,  and  Marine 
Corps  program  to  develop  and  field  stealthy  fighter  aircraft  to  replace  DOD’s  aging  fighter 
and  attack  aircraft.  FCS  is  an  Army  program  intended  to  provide  advanced,  networked 
combat  and  sustainment  systems;  unmanned  ground  and  air  vehicles;  and  unattended 
sensors  and  munitions.  WIN-T  is  intended  to  provide  the  Army  with  a  high-speed,  high- 
capacity  communications  network.  MMA  is  a  Navy  program  intended  to  provide  persistent 
antisubmarine  and  antisurface  warfare  and  intelligence,  surveillance,  and  reconnaissance 
capabilities. 

3  GAO,  Cost  Assessment  Guide:  Best  Practices  for  Estim  ating  and  Managing  Program 
Costs,  GAO-07-1134SP  (Washington,  D.C.:  July  2007)- 
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evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained 
provides  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our 
audit  objectives. 


Results  in  Brief 


DOD  often  does  not  commit  full  funding  to  its  major  weapon  system 
acquisition  programs,  despite  the  department’s  policy  to  “fully  fund” 
programs  at  the  start  of  system  development.4  Of  the  programs  we 
reviewed,  over  75  percent  were  not  fully  funded  in  the  Future  Years 
Defense  Program  (FYDP) — DOD’s  investment  strategy.  We  found  that 
because  programs  typically  had  development  cycles  that  extended  beyond 
the  FYDP  time  frame,  the  FYDP  did  not  capture  their  full  funding  needs.  At 
the  same  time,  program  costs  exceeded  the  funding  levels  initially  planned 
for  and  reflected  in  the  years  covered  by  the  FYDP.  To  compensate  for 
funding  shortfalls,  DOD  often  makes  unplanned  and  inefficient  funding 
adjustments,  such  as  moving  money  between  programs,  deferring  work 
and  associated  costs  into  the  future,  or  reducing  procurement  quantities. 
Ultimately,  such  reactive  practices  obscure  true  program  costs  and 
contribute  to  the  instability  of  many  programs  and  poor  acquisition 
outcomes. 

DOD’s  inability  to  allocate  funding  effectively  to  programs  is  largely  driven 
by  the  acceptance  of  unrealistic  cost  estimates  and  a  failure  to  balance 
needs  based  on  available  resources.  Development  costs  for  major 
acquisition  programs  are  often  underestimated  at  program  initiation — 30 
to  40  percent  in  some  cases — in  large  part  because  the  estimates  are  based 
on  limited  knowledge  and  optimistic  assumptions  about  system 
requirements  and  critical  technologies.  For  example,  initial  development 
cost  estimates  for  the  Army’s  WIN-T  communications  system  were 
understated  by  at  least  $1.3  billion,  or  nearly  160  percent,  in  part  because 
the  estimates  assumed  that  commercial-off-the-shelf  radio  technology 
would  be  available.  This  assumption  proved  to  be  wrong.  Similarly,  JSF’s 
development  costs  were  underestimated  by  at  least  $7.1  billion,  or  around 
20  percent,  because  the  initial  estimate  assumed  certain  efficiencies  that 
never  materialized.  These  unrealistic  cost  estimates  are  developed  in  an 


4  Full  funding  is  a  DOD  5000.2  requirement  for  formal  program  initiation  of  an  acquisition 
program.  In  this  sense,  full  funding  means  having  an  approved  current  (and  projected) 
resource  stream  to  execute  the  acquisition  program;  that  is,  program  funding  is  included 
both  in  the  budget  and  in  the  out-years  of  the  Future  Years  Defense  Program  sufficient  to 
cover  the  current  and  future  efforts  described  in  the  acquisition  strategy. 
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environment  where  DOD  commits  to  more  programs  than  available 
resources  can  support,  which  promotes  unhealthy  competition  among 
programs  for  funding.  This  competition  creates  strong  incentives  for 
program  officials  to  establish  requirements  that  make  their  particular 
weapon  systems  stand  out  from  others,  with  less  consideration  given  to 
the  resources  that  will  be  needed  to  develop  them.  Ultimately,  programs 
tend  to  push  the  need  for  funding  to  the  future  rather  than  limit  program 
length  or  adjust  requirements. 

Our  past  work  on  commercial  best  practices  found  that  successful 
companies  were  able  to  fully  fund  their  product  development  programs 
because  they  required  programs  to  have  realistic,  knowledge-based  cost 
estimates;  they  committed  to  manageable  product  increments;  and  they 
prioritized  programs  within  resource  constraints.  To  develop  realistic  cost 
estimates,  these  companies  followed  a  disciplined  process  through  which 
requirements  were  assessed  against  available  resources  at  multiple  gated 
reviews  prior  to  committing  to  a  new  product.  As  more  knowledge  was 
gained  and  risk  was  reduced,  costs  estimates  more  accurately  reflected 
true  costs.  Before  initiating  product  development,  for  example,  Motorola 
and  IBM  expected  that  actual  costs  would  not  exceed  the  latest  estimate 
by  more  than  5  or  10  percent.  To  increase  the  predictability  of  funding 
needs — and  funding  stability — these  companies  expected  development 
cycles  to  be  manageable  (2  to  5  years).  Likewise,  a  study  commissioned  by 
DOD  in  2006  recommended  that  programs  should  be  time-constrained 
with  development  cycles  no  longer  than  6  years,  and  DOD  is  taking  steps 
to  pilot  this  concept.  Finally,  companies  used  a  portfolio  management 
approach  to  prioritize  investments  and  allocate  resources.  Such  an 
approach  requires  tough  decisions  about  approving  or  terminating 
programs.  Once  programs  were  approved,  these  companies  made  a  firm 
commitment  to  fully  fund  them.  Ultimately,  these  practices  helped 
companies  avoid  committing  to  more  programs  than  they  could  afford  and 
allowed  them  to  optimize  the  return  on  their  investments. 

To  more  effectively  fund  major  weapon  system  acquisition  programs  and 
achieve  successful  outcomes,  we  are  recommending  that  the  Secretary  of 
Defense  develop  and  implement  a  strategy  to  bring  DOD’s  current  weapon 
systems  portfolio  into  balance  by  aligning  the  number  of  systems  with 
available  resources  in  the  FYDP.  In  addition,  the  Secretary  should  require 
new  programs  to  have  manageable  development  cycles,  realistic  cost 
estimates,  and  to  have  planned  and  programmed  full  funding  for  the  entire 
development  cycle.  Finally,  the  Secretary  should  require  all  cost  estimates 
submitted  for  funding  a  program  at  milestone  decisions  to  be  reported  as  a 
range  of  likely  costs  and  reflect  the  associated  levels  of  risk  and 
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uncertainty.  DOD  partially  concurred  with  our  first  two  recommendations, 
but  believes  that  they  are  being  sufficiently  addressed  through  current 
acquisition  policies  and  ongoing  initiatives.  While  we  agree  that  if 
implemented  appropriately,  DOD’s  current  policies  and  initiatives  have  the 
potential  to  contribute  to  better  program  outcomes,  we  have  found  no 
evidence  of  widespread  adoption  of  a  knowledge-based  process  to  better 
ensure  adequate  funding  for  programs.  DOD  agreed  with  the  spirit  of  our 
third  recommendation — to  consider  risks  at  key  milestones — however,  it 
did  not  agree  that  all  cost  estimates  submitted  for  funding  a  program  at 
milestone  decisions  should  be  reported  as  a  range  of  likely  costs.  In  its 
written  comments,  DOD  stated  that  a  certain  method  used  to  calculate 
cost  ranges  can  produce  misleading  results.  While  we  do  not  advocate  a 
specific  method  for  calculating  a  range  of  potential  costs,  we  maintain  that 
presenting  such  a  range  would  provide  decision  makers  with  additional 
knowledge  about  the  level  of  risk  in  a  proposed  program. 


Background 


To  plan,  execute,  and  fund  its  weapon  system  acquisition  programs,  DOD 
relies  on  three  principal  decision-making  systems:  the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS),  which  is  used  to  assess 
gaps  in  warfighting  capabilities  and  recommend  solutions  to  resolve  those 
gaps;  the  Defense  Acquisition  System  (DAS),  which  is  used  to  manage  the 
development  and  procurement  of  weapon  systems  and  other  equipment; 
and  the  Planning,  Programming,  Budgeting,  and  Execution  (PPBE) 
process,  which  is  used  to  allocate  resources.  While  the  JCIDS  and  DAS 
processes  are  driven  by  specific  events — such  as  validating  requirements 
or  receiving  approval  to  start  development — the  PPBE  process  is  calendar 
driven,  taking  nearly  2  years  to  go  from  planning  to  the  beginning  of 
budget  execution. 

The  PPBE  process  is  intended  to  provide  a  framework  within  which  DOD 
can  articulate  its  strategy;  identify  force  size,  structure,  and  needed 
equipment;  set  program  priorities;  allocate  resources  to  individual 
programs;  and  assess  program  performance.  Although  the  different  phases 
of  PPBE  are  considered  sequential,  because  of  the  amount  of  time 
required  to  develop  and  review  resource  requirements,  the  process  is 
continuous  and  concurrent  with  at  least  two  phases  ongoing  at  any  given 
time  (see  fig.  1). 
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Figure  1:  Simplified  View  of  PPBE  Process 


Source:  DOD  (data);  GAO  (analysis  and  presentation). 


At  the  front  end  of  the  PPBE  process,  the  Secretary  of  Defense  provides 
planning  guidance  to  the  military  services  and  defense  agencies  about  the 
capabilities  and  resources  required  to  deter  and  defeat  threats.  The 
services  and  agencies,  in  turn,  develop  individual  “programs”  (budgets) 
based  on  the  planning  guidance  as  well  as  fiscal  guidance  provided  by  the 
Office  of  Management  and  Budget  (OMB).  The  service  and  agency 
program  budgets  are  then  subjected  to  a  series  of  leadership  reviews  in  the 
department — OSD,  Director  of  Program  Analysis  and  Evaluation,  Joint 
Chiefs  of  Staff,  and  the  Comptroller — and  adjustments  are  made  if 
necessary.  Responsibility  for  managing  the  PPBE  process  and  ensuring  the 
budget  is  prepared  and  submitted  to  Congress  resides  with  the  OSD 
Comptroller. 

The  PPBE  process  produces  the  defense  portion  of  the  President’s  annual 
budget  request  to  Congress  as  well  as  the  FYDP — DOD’s  longer-term 
investment  strategy — which  includes  the  department’s  resource 
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allocations  for  the  4  to  5  fiscal  years  beyond  the  budget.6  The  ultimate 
objective  of  the  FYDP  is  to  manage  funds  in  a  way  that  provides 
combatant  commanders  with  the  best  mix  of  forces,  equipment,  and 
support  attainable  within  fiscal  constraints  established  by  OMB.  Once 
complete,  the  FYDP  is  expected  to  provide  an  aggregate  picture  of  the 
anticipated  force  levels  and  funding  needs  of  individual  programs.  Data  in 
the  FYDP  tell  the  Secretary  of  Defense,  the  President,  Congress,  and  the 
American  people  what  the  department  expects  to  invest  in  and  how  much 
it  will  spend  over  time. 

DOD  policy  requires  that  the  dollars  and  manpower  needed  to  carry  out 
the  first  5-  to  6-years  of  a  weapon  system  acquisition  program  must  be 
included  in  the  FYDP.  The  value  of  the  FYDP  greatly  depends  on  the 
accuracy  of  the  cost  estimates  supporting  each  individual  program.  Having 
a  realistic  cost  estimate  provides  a  basis  for  accurate  budgeting  and 
effective  resource  allocation,  increasing  the  probability  of  a  program’s 
success  in  meeting  its  targets.  Two  cost  estimates  are  required  before  a 
program  is  approved  to  start  system  development  (Milestone  B)  and  at  the 
beginning  of  production  (Milestone  C) — the  service  or  program  office 
estimate  and  an  independent  cost  estimate.  The  independent  cost  estimate 
for  most  major  acquisition  programs  is  developed  by  the  Cost  Analysis 
Improvement  Group  (CAIG).  At  Milestone  B,  the  Milestone  Decision 
Authority  (MDA)6  uses  the  two  estimates  to  determine  the  program’s 
official  cost  baseline  and  whether  the  funding  reflected  in  the  FYDP  is 
adequate  to  cover  the  portion  of  the  estimated  costs  of  the  program  that 
fall  within  the  FYDP  time  period.7  If  not,  the  services  are  expected  to 
adjust  the  FYDP  to  ensure  that  their  programs  are  fully  funded  to  the 
approved  baseline.  To  receive  Milestone  B  approval,  DOD  is  required  to 


5  DOD  modified  the  Planning,  Programming,  and  Budgeting  System  (PPBS)  process, 
renamed  the  PPBE,  in  2003.  PPBE  in  its  entirety  is  not  implemented  every  year  even  though 
DOD  must  request  funding  from  Congress  annually.  Full-scale  planning  and  programming 
activities  are  conducted  in  even-numbered  years  (called  on-years)  while  programming 
adjustments  are  made  in  odd-numbered  years  (called  off-years).  As  a  result  of  this 
modification,  the  number  of  years  covered  by  the  FYDP  now  alternates  between  5  years 
(the  President’s  budget  plus  4  years)  and  6  years  (the  President’s  budget  plus  5  years). 

6  The  MDA  for  all  major  defense  acquisition  programs  is  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  unless  a  program  is  delegated  to  the  head  of  a  DOD 
component. 

'  Although  DOD  policy  requires  full  funding  within  the  FYDP,  most  major  weapon  system 
acquisition  programs  have  development  cycles  that  extend  beyond  the  FYDP  time  period. 
However,  there  is  no  requirement  to  constrain  development  cycles  to  fit  within  that  same 
time  period. 
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certify  to  Congress 8  that  funding  is  available  to  support  the  portion  of  the 
program  that  falls  within  the  period  covered  by  the  FYDP  that  is  submitted 
during  the  fiscal  year  in  which  the  certification  is  made.9 

Our  past  work  has  consistently  shown  that  DOD’s  major  weapon  system 
programs  do  not  meet  their  cost  and  schedule  targets.  Since  the  mid-1990s, 
we  have  studied  leading  commercial  companies  in  order  to  identify  best 
practices  for  developing  and  producing  new  products.  Taking  into  account 
the  differences  between  commercial  product  development  and  weapons 
acquisitions,  we  articulated  a  best  practices  model  that  relies  on 
increasing  knowledge  when  developing  new  products,  separating 
technology  development  from  product  development,  and  following  an 
evolutionary  or  incremental  approach  to  product  development.10  This 
knowledge-based  approach  requires  developers  to  make  investment 
decisions  on  the  basis  of  specific,  measurable  levels  of  knowledge  at 
critical  junctures  before  investing  more  money  and  before  advancing  to 
the  next  phase  of  acquisition.  An  evolutionary  product  development 
process  defines  the  individual  increments  on  the  basis  of  mature 


8  U.S.C.  Title  10  §  2366(a),  which  was  enacted  in  2006,  specifies  that  a  major  defense 
acquisition  program  may  not  receive  Milestone  B  approval,  or  Key  Decision  Point  B 
approval  in  the  case  of  a  space  program,  until  the  MDA  certifies  that  (1)  the  program 
technology  has  been  demonstrated  in  a  relevant  environment;  (2)  the  program 
demonstrates  a  high  likelihood  of  accomplishing  its  mission;  (3)  the  program  is  affordable 
when  considering  the  per  unit  cost  and  the  total  acquisition  cost  in  the  context  of  the  total 
resources  available  during  the  period  covered  by  the  FYDP  submitted  during  the  fiscal  year 
in  which  the  certification  is  made;  (4)  the  department  has  completed  an  analysis  of 
alternatives;  (5)  the  program  is  affordable  when  considering  the  ability  of  DOD  to 
accomplish  the  program’s  mission  using  alternative  systems;  (6)  the  Joint  Requirements 
Oversight  Council  has  accomplished  its  duties  with  respect  to  the  program  pursuant  to 
section  181  (b)  of  this  title,  including  an  analysis  of  the  operational  requirements  for  the 
program;  and  (7)  the  program  complies  with  all  relevant  policies,  regulations,  and 
directives  of  DOD. 

9  Programs  are  not  required  to  hold  a  Milestone  A  review,  the  point  at  which  concept 
refinement  ends  and  technology  development  begins.  However,  in  2007,  Congress  enacted 
legislation  specifying  that  a  major  defense  acquisition  program  beginning  after  March  2008 
may  not  receive  Milestone  A  approval,  to  begin  a  technology  development  program,  until 
the  MDA  certifies  to  Congress  that  (1)  the  system  fulfills  an  approved  initial  capabilities 
document;  (2)  the  system  is  being  executed  by  an  entity  with  a  relevant  core  competency 
as  identified  by  the  Secretary  of  Defense;  (3)  if  the  system  duplicates  a  capability  already 
provided  by  an  existing  system,  the  duplication  provided  by  such  system  is  necessary  and 
appropriate;  and  (4)  a  cost  estimate  for  the  system  has  been  submitted.  DOD  is  currently 
revising  its  policy  and  guidance  for  conducting  and  certifying  Milestone  A  reviews.  National 
Defense  Authorization  Act  for  Fiscal  Year  2008,  P.L.  No.  110-181,  §  943,  122  Stat.  3,  288-89 
(2008). 

10  See  Related  GAO  Products  at  the  end  of  this  report. 
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technologies  and  a  feasible  design  that  are  matched  with  firm 
requirements.  The  knowledge-based,  evolutionary  approach  in  our  model 
is  intended  to  help  reduce  development  risks  and  to  achieve  better 
program  outcomes  on  a  more  consistent  basis. 

In  October  2000,  DOD  began  to  significantly  revise  the  department’s 
acquisition  policy,  adopting  a  knowledge-based,  evolutionary  product 
development  approach.  DOD’s  revised  policy  emphasizes  the  importance 
of  and  provides  a  good  framework  for  capturing  knowledge  about  critical 
technologies,  product  design,  and  manufacturing  processes.  In  recent 
years,  we  have  reported  that  if  properly  implemented  and  enforced,  DOD’s 
acquisition  policy  could  reduce  technical  risk  at  the  start  of  a  program  and 
make  cost  and  delivery  estimates  much  more  predictable. 


Failure  to  Commit 
Full  Funding  to 
Weapon  Systems 
Contributes  to  Poor 
Acquisition  Outcomes 


DOD  often  does  not  commit  full  funding  to  its  major  weapon  system 
acquisitions  when  they  are  initiated,  despite  the  department’s  policy  to  do 
so.  For  a  majority  of  the  programs  we  reviewed,  costs  exceeded  the 
funding  levels  initially  planned  for  and  reflected  in  the  FYDP.  To  make  up 
for  these  funding  shortfalls,  DOD  often  shifts  funds  from  one  program  to 
pay  for  another,  reduces  system  capabilities,  cuts  procurement  quantities, 
extends  development  and  procurement  schedules,  or  in  rare  cases 
terminates  programs.  Such  actions  not  only  create  instability  in  DOD’s 
weapon  system  portfolio,  they  also  obscure  the  true  future  costs  of  current 
commitments,  making  it  difficult  to  make  informed  investment  decisions. 


At  the  beginning  of  system  development,  the  approved  cost  estimates  for 
most  of  the  major  acquisition  programs  we  reviewed  have  exceeded  the 
funding  initially  established  in  the  FYDP.  First,  our  analysis  indicates  that 
14  of  the  20  programs  had  development  cycles  that  extended  beyond  the 
FYDP.  As  a  result,  the  FYDP  could  not  capture  all  of  the  estimated  funding 
needed  to  complete  development.  Second,  the  funding  established  in  the 
FYDP  at  the  beginning  of  system  development  for  15  of  the  20  programs 
was  less  than  what  they  required  for  that  time  period.11  For  example,  at  the 
start  of  system  development  in  2003,  the  initial  approved  cost  estimate  for 


11  The  funding  needed  within  the  FYDP  for  each  program  was  calculated  using  the  funding 
data  contained  in  the  Research,  Development,  Test,  and  Evaluation  Annual  Funding 
Summary  of  the  first  selected  acquisition  report  issued  after  system  development  began. 
The  amount  of  funding  established  in  the  FYDP  was  calculated  using  each  program’s 
RDT&E  budget  justification  documentation  for  the  year  in  which  system  development  was 
initiated. 
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the  development  portion  of  the  FCS  program  was  about  $20  billion,  of 
which  $13  billion  was  needed  within  the  2003  FYDP  time  frame — that  is 
for  fiscal  years  2003  through  2007.  However,  at  that  time  the  2003  FYDP 
only  contained  $5  billion  for  FCS  development.  This  means  that  the 
department  had  only  committed  to  fund  25  percent  of  FCS’s  estimated 
development  cost  when  the  program  started,  and  only  39  percent  of  the 
funding  needed  in  its  first  5  years.  Other  programs  we  reviewed 
experienced  similar  shortfalls  (see  fig.  2). 


Figure  2:  Shortfalls  at  the  Start  of  Development  for  Five  Major  Weapon  System 
Programs 


0  10  20  30  40  50  60  70  80  90  100 

Percentage  development  funding 


| _ |  Level  of  funding  in  the  FYDP  in  the  year  the  program  was  initiated 

Level  of  funding  the  program  needed  to  be  fully  funded  in  the  initial  FYDP 
Funding  required  beyond  the  initial  FYDP  to  complete  development 
Source:  DOD  (data);  GAO  (analysis  and  presentation). 


To  bring  funding  commitments  more  in  line  with  original  program 
baselines,  DOD  typically  makes  adjustments  in  subsequent  FYDPs.  By  that 
time,  however,  program  cost  baselines  have  often  increased — as  more 
knowledge  about  technologies,  design,  and  manufacturing  needs  is 
gained — creating  new  funding  shortfalls.  For  example,  by  the  third  year  of 
the  WIN-T  program,  the  program’s  cost  estimate,  and  therefore  funding 
needs,  had  increased  72  percent  because  requirements  were  unstable  and 
critical  technologies  were  not  ready,  creating  a  26  percent  shortfall  in  the 
FYDP.  Similarly,  a  57  percent  increase  in  FCS’s  cost  estimate  created  an 
additional  12  percent — or  $2  billion — funding  shortfall. 
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In  reaction  to  funding  shortfalls  in  individual  programs,  DOD  often  moves 
funds  from  other  programs,12  reduces  capabilities  by  scaling  back 
requirements,  cuts  procurement  quantities,  or  extends  development 
schedules.  Although  rare,  programs  have  also  been  terminated.  Such 
reactive  measures  destabilize  programs  and  in  many  cases  increase 
program  costs.  For  example,  as  we  previously  reported,  funding  for  the  Air 
Force’s  Space-Based  Infrared  System-High  (SIBRS)  satellite  system  was 
cut  in  1998  and  1999  to  pay  for  higher  budget  priorities,  which  contributed 
to  a  2-year  delay  in  the  program  and  a  breach  of  the  cost  baseline.  As  a 
result,  the  Air  Force  changed  SIBRS’s  procurement  strategy,  which 
independent  cost  estimators  calculated  would  double  program  costs.13  For 
the  F-22A  Raptor  program,  our  past  work  has  noted  that  the  Air  Force 
drastically  reduced  its  planned  buys  from  648  to  183  as  program  costs 
escalated.  Similarly,  the  number  of  requirements  for  the  Joint  Tactical 
Radio  System  (JTRS)  was  reduced  or  deferred  by  about  one-third  as  the 
program  encountered  development  problems.  This  change  had  a 
reverberating  effect  on  several  JTRS-dependent  efforts — such  as  the 
Army’s  modernization  of  radios  for  its  helicopters — as  those  programs  had 
to  make  adjustments  and  go  forward  with  alternative,  less  capable 
solutions.14  Making  these  types  of  adjustments  to  compensate  for 
inadequate  funding  obscures  future  resource  requirements  and  limits  the 
Congress’s  visibility  over  DOD’s  true  funding  needs. 


12  DOD  has  a  formal  process  for  reprogramming  appropriated  funds,  including  if  necessary, 
congressional  notification  and  approval. 

13  GAO,  Space  Acquisitions:  DOD  Needs  to  Take  More  Action  to  Address  Unrealistic 
Initial  Cost  Estimates  for  Space  Systems,  GAO-07-96  (Washington,  D.C.:  Nov.  17,  2006). 

14  GAO-07-388. 
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Unrealistic  Cost 
Estimates  and  a 
Failure  to  Balance 
Needs  with  Available 
Resources  Underlie 
DOD’s  Flawed 
Funding  Approach 


For  most  of  the  20  major  acquisition  programs  that  we  reviewed,  initial 
funding  was  based  on  cost  estimates  that  proved  to  be  too  low.  DOD’s 
funding  approach  is  further  compromised  by  the  department’s  failure  to 
balance  requirements  with  available  resources.  Because  DOD  commits  to 
more  programs  than  resources  can  support,  programs  often  have  to 
compete  for  funding  by  overpromising  capabilities  and  by  providing  low 
cost  estimates  as  inputs  to  the  funding  process.  We  have  previously 
reported  that  when  such  trends  go  unchecked,  Congress  is  consistently 
faced  with  a  difficult  choice:  pull  funds  from  other  federal  programs  to 
support  DOD’s  acquisitions  or  accept  less  warfighting  capability  than 
promised.15 


Unrealistic  Cost  Estimates  The  foundation  of  an  accurate  funding  commitment  for  a  weapon  system 
Drive  Inaccurate  Funding  program  should  be  a  realistic  cost  estimate  that  is  based  on  a  high  degree 

Commitments  °f  knowledge  about  requirements,  technology,  design,  and  manufacturing. 

Realistic  estimates  also  provide  a  sound  basis  for  setting  priorities  by 
allowing  decision  makers  to  compare  the  relative  value  of  one  program  to 
another  and  to  make  adjustments  accordingly.  These  adjustments  could 
include  a  decision  about  whether  to  proceed  with  a  program,  reduce 
requirements,  or  defer  requirements  to  a  future  increment.  Most  of  the  20 
programs  we  reviewed  underestimated  costs.  Unrealistic  cost  estimates 
are  largely  the  result  of  a  lack  of  knowledge,  failure  to  adequately  account 
for  risk  and  uncertainty,  and  overly  optimistic  assumptions  about  the  time 
and  resources  needed  to  develop  weapon  systems.  By  repeatedly  relying 
on  unrealistically  low  cost  estimates,  DOD  has  initiated  more  programs 
than  its  budget  can  support. 

Our  assessment  of  cost  data  for  20  major  acquisition  programs  in  DOD’s 
current  portfolio  found  that  the  majority  of  these  programs  were  initiated 
with  cost  estimates  for  system  development  that  were  too  low — a  finding 
consistent  with  our  prior  work  and  with  cost  growth  patterns  reported  by 
RAND,  the  Institute  for  Defense  Analysis  (IDA),  and  other  organizations 
that  conduct  defense  analyses.16  For  19  of  the  20  programs,  the 


15  GAO-07-388. 

16  Assessment  Panel  of  the  Defense  Acquisition  Performance  Assessment  Project  for  the 
Deputy  Secretary  of  Defense,  Defense  Acquisition  Performance  Assessment  Report 
(Washington,  D.C.:  Jan.  2006);  Defense  Science  Board,  Defense  Science  Board  Summer 
Study  on  Transformation:  A  Progress  Assessment  (Washington,  D.C.:  Feb.  2006);  RAND, 
Historical  Cost  Growth  of  Completed  Weapon  System  Programs  (Santa  Monica:  2006);  and 
Institute  for  Defense  Analysis,  Costs  Growth  in  Major  Weapon  Procurement  Programs, 
Presentation  to  the  38th  Annual  DoD  Cost  Analysis  Symposium  (Williamsburg:  Feb.  2005). 


Page  12 


GAO-08-619  Defense  Acquisitions 


independent  CAIG  estimates  were  higher  than  the  service  estimates — by 
as  much  as  139  percent  in  one  case — yet  the  CAIG  estimates  for  5  of  those 
programs  were  still  understated  by  billions  of  dollars  (see  table  1).  In 
addition,  while  5  of  the  20  programs  had  not  reported  cost  growth  as  of 
December  2007,  the  remaining  15  programs  had.  For  example,  the  initial 
CAIG  estimate  for  the  Expeditionary  Fighting  Vehicle  (EFV)  program  was 
about  $1.4  billion  compared  to  a  service  estimate  of  about  $1.1  billion,  but 
development  costs  for  the  EFV  system  is  now  expected  to  be  close  to 
$3.6  billion.  Similarly,  the  Army  initially  estimated  that  WIN-T  development 
would  cost  $338  million,  but  the  development  program  is  now  expected  to 
cost  over  $2.0  billion,  or  $1.7  billion  more  than  initially  estimated.  Eight  of 
the  20  programs  have  reported  development  cost  growth  of  more  than  35 
percent,  resulting  in  the  need  for  nearly  $19  billion  in  additional  funding. 
Estimates  that  are  this  far  off  the  mark  do  not  provide  the  necessary 
foundation  for  sufficient  funding  commitments. 
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Table  1:  Development  Cost  Estimates  and  Baselines  for  20  Major  Weapon  System  Programs 


(2008  dollars  in  millions) 


Development  cost 
estimate 

Development  cost 
baselines 

Program 

Service 

CAIG 

Percentage 

difference 

Initial 

Current 

Percentage 

change 

Global  Hawk 

$905 

$992 

10 

$967 

$3,515 

264 

UH-60M  helicopter  upgrade 

311 

379 

22 

311 

838 

169 

WIN-T 

338“ 

807 

139 

796 

2,088 

162 

C-130  Avionics  Modernization 
Program 

1,020 

1,175 

15 

720 

1,844 

156 

EFV 

1,056 

1,438 

36 

1,472 

3,556 

142 

Advanced  Extremely  High 

Frequency  Satellites 

3,031 

3,175 

5 

2,923 

6,008 

105 

Wideband  Global  SATCOM 

296 

414 

40 

199 

323 

62 

Future  Combat  Systems 

20,248 

27,184 

34 

20,248 

27,955 

38 

JSFb 

30,500 

31,476 

3 

33,939 

40,210 

18 

COBRA  JUDY  replacement 

1,398 

1,521 

9 

1,527 

1,626 

6 

E-2  Advanced  Hawkeye 

3,495 

3,720 

6 

3,589 

3,796 

6 

EA-18G 

1,707 

1,795 

5 

1,797 

1,865 

4 

VH-71  Presidential  Helicopter 
Replacement  Program 

3,378 

3,569 

6 

3,771 

3,859 

2 

Joint  Land  Attack  Cruise  Missile 
Defense  Elevated  Netted  Sensor 

1,781 

1,926 

8 

1,894 

1,922 

1 

C-5  Reliability  Enhancement  and 
Reengining  Program 

1,454 

1,583 

9 

1,627 

1,630 

0° 

CH-53K  Heavy  Lift  Replacement 

3,970 

4,293 

8 

4,149 

4,095 

-1 

Longbow  Apache  III 

1,155 

1,382 

20 

1,095 

1,087 

-1 

MMA 

6,100 

6,970 

14 

7,080 

6,804 

-4 

Small  Diameter  Bomb 

416 

427 

3 

415 

395 

-5 

Standard  Missile  6 

1,000 

992 

-1 

1,009 

932 

-8 

Source:  GAO  analysis  of  DOD  data. 

“The  service’s  original  estimate  for  the  WIN-T  program  of  $338  million  was  revised  prior  to  Milestone 
B  to  align  with  the  CAIG  estimate. 

bJSF  data  include  Air  Force  and  Navy  portions  of  the  program  only. 

cC-5  RERP  costs  have  increased  by  almost  $4  million.  Flowever,  due  to  rounding,  the  table  indicates 
a  0  percent  increase. 
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DOD  has  found  similar  problems  in  other  acquisition  programs.  For 
example,  in  2003,  a  DOD  study  found  that  space  programs  are  strongly 
biased  to  produce  unrealistically  low  cost  estimates  throughout  the 
acquisition  process.17  The  study  found  that  most  programs  at  the  time  of 
contract  initiation  had  a  predictable  cost  growth  of  50  to  100  percent.  The 
study  also  found  that  the  unrealistically  low  projections  of  program  cost 
and  lack  of  provisions  for  management  reserve  seriously  distorted 
management  decisions  and  program  content,  increased  risks  to  mission 
success,  and  virtually  guaranteed  program  delays. 

Inaccurate  cost  estimates  are  often  the  result  of  limited  knowledge  about 
requirements  and  technologies.  Our  best  practices  work  has  shown  that 
conducting  early  disciplined  analysis,  such  as  systems  engineering,18  builds 
knowledge  that  enables  a  developer  to  identify  and  resolve  gaps  between 
requirements  and  available  resources  before  beginning  product 
development.  DOD’s  acquisition  policy  and  guidance  emphasize  the 
importance  of  obtaining  knowledge  prior  to  Milestone  B — the  start  of 
system  development — through  key  analyses  and  activities — such  as 
conducting  an  analysis  of  alternatives  (AO A), 19  refining  requirements,  and 
reducing  technology  risks.  Knowledge  gained  from  these  analyses  and 
activities  should  inform  a  Cost  Analysis  Requirements  Description 
(CARD) — a  key  document  that  quantifies  the  program’s  technical, 
physical,  programmatic,  and  performance  characteristics  for  developing 
cost  estimates.  However,  we  have  frequently  reported  that  DOD  programs 
do  not  adequately  define  requirements,  mature  technologies,  or  develop  an 
effective  acquisition  strategy  before  Milestone  B — the  point  at  which  cost 
estimates  are  approved.20  According  to  several  CAIG  analysts,  CARDs 


17  Defense  Science  Board,  Report  of  the  Defense  Science  Board  /  Air  Force  Advisory 
Board,  Joint  Task  Force  on  Acquisition  of  National  Security  Space  Programs 
(Washington,  D.C.:  May  2003). 

18  Systems  engineering  is  a  technical  management  tool  that  provides  the  knowledge 
necessary  to  translate  requirements  into  specific,  achievable  capabilities.  By  using  the  tools 
of  systems  engineering  during  the  early  phases  of  concept  refinement  and  technology 
development,  acquisition  decision  makers  and  developers  can  work  together  to  close  gaps 
between  requirements  and  available  resources  well  before  system  development  starts. 

19  An  AOA  should  compare  the  costs  and  benefits  of  alternative  solutions  for  meeting  a 
validated  need.  As  such  an  AOA  should  assess  the  life  cycle  cost,  schedule,  and  operational 
effectiveness  of  each  possible  solution  and  identify  a  preferred  alternative. 

20  GAO,  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs, 
GAO-08-467SP  (Washington,  D.C.:  Mar.  31,  2008),  and  Defense  Acquisitions:  Major  Weapon 
Systems  Continue  to  Experience  Cost  and  Schedule  Problenrns  under  DOD’s  Revised 
Policy,  GAO-06-368  (Washington,  D.C.:  Apr.  13,  2006). 
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often  lack  sufficient  detail  about  planned  program  content  to  develop 
sound  cost  estimates.  One  analyst  noted  that  the  CARD  for  the  FCS 
program  had  to  be  sent  back  to  the  program  office  because  it  was  too 
vague. 

At  the  same  time,  programs  are  expected  to  deliver  the  CARD  to  the  CAIG 
at  least  180  days  prior  to  Milestone  B,  while  other  supporting  documents 
that  contain  critical  program  information — such  as  the  program 
requirements  document — are  not  required  until  about  90  days  in  advance 
of  a  milestone  review.  Many  programs  conduct  analyses  of  alternatives 
and  other  analyses  at  the  same  time  program  cost  estimates  are  being 
developed.  According  to  a  senior  CAIG  official,  such  concurrency  limits 
their  ability  to  develop  a  realistic  independent  estimate.  One  senior  DOD 
official  also  noted  that  the  department’s  acquisition  policy  may  have 
encouraged  programs  to  rush  to  Milestone  B,  limiting  the  quality  of  the 
data  available  to  develop  a  program  cost  estimate.  Although  DOD’s 
acquisition  policy  has  included  an  early  formal  review  at  Milestone  A,  this 
review  was  not  mandatory,  and  as  we  have  reported  in  the  past,  most 
major  acquisition  programs  have  not  gone  through  this  early  review.21 

Cost  analysts  have  also  indicated  that  the  lack  of  comparable  products  and 
good  historical  program  data  limits  knowledge,  making  it  difficult  to 
develop  realistic  cost  estimates.  This  is  true  for  the  majority  of  DOD’s 
larger  and  more  complex  transformational  programs  that  are  seeking  to 
provide  capabilities  that  require  advanced  technologies  and  equipment 
that  have  no  historical  precedent,  like  JSF  and  FCS. 

In  the  absence  of  knowledge,  cost  estimators  must  rely  heavily  on 
assumptions  about  system  requirements,  technology,  and  design  maturity 
as  well  as  the  time  and  funding  needed.  However,  the  level  of  resources 
needed  for  product  development  is  often  understated,  as  we  found  in 
several  of  the  programs  we  reviewed.  For  example: 

•  FCS:  The  estimated  lines  of  code  needed  to  support  FCS’s  software 
development  are  almost  three  times  original  assumptions — from  32 
million  to  95  million  lines  of  code — leading  to  an  increase  in  software 
development  costs  that  now  approaches  $8  billion.  A  2007  IDA  report 
also  identified  significant  additional  unquantifiable  cost  risk  due  to 
immature  technologies,  dependencies  on  complementary  programs, 


21  GAO-06-368  and  GAO-07-388. 
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concurrent  experimentation  and  development,  and  the  overall 
complexity  and  synchronization  of  FCS  development  activities.22 

•  JSF:  JSF  assumed  that  the  commonality  between  the  three  variants  of 
aircraft  and  the  use  of  a  joint  development  program,  instead  of  three 
separate  programs,  could  cut  development  costs  by  about  40  percent. 
However,  after  development  started,  significant  design  issues  forced 
the  program  to  delay  development  approximately  18  months  to 
conduct  unexpected  design  work.  In  addition,  the  assumed 
commonality  between  the  variants  decreased.  As  we  reported  in  2005, 
these  two  factors  contributed  to  cost  increases  that  nearly  eroded  all  of 
the  assumed  cost  savings.23 

•  WIN-T:  The  Army  assumed  that  the  radios  and  software  needed  to 
support  the  WIN-T  system  would  be  commercially  available.  However, 
once  system  development  started,  the  Army  learned  that  the  radios  and 
software  would  require  significantly  more  development  and  integration 
than  initially  anticipated.  Further,  the  Army  assumed  that  WIN-T  would 
be  able  to  meet  its  portion  of  the  FCS  program  requirements.  However, 
subsequent  changes  in  the  FCS  requirements  contributed  to  the  need  to 
restructure  the  WIN-T  program. 

•  Global  Hawk:  The  Air  Force  assumed  that  validated  warfighter 
requirements  could  be  met  with  minor  additional  development  to  a 
smaller  version,  subsequently  designated  the  RQ-4A.  However,  1  year 
after  initiating  both  system  development  and  low-rate  initial 
production,  it  was  determined  that  a  larger  airframe  and  additional, 
unproven  technologies  would  be  needed.  As  a  result,  DOD  restructured 
the  acquisition  strategy  to  include  a  second  model — designated  RQ- 
4B — tripling  development  costs  and  extending  the  development  cycle 
from  7  years  to  12  years. 

These  examples  are  consistent  with  our  work  over  the  past  several 
decades,  which  has  raised  concerns  about  the  cost  and  schedule 
implications  of  DOD’s  cost  estimating  assumptions  in  its  major 
acquisitions.  Our  work  on  DOD’s  space  acquisition  programs  has  found 


22  Institute  for  Defense  Analysis,  Future  Combat  Systems  (FCS)  Cost  Review:  Summary  of 
Findings  (Alexandria:  Apr.  2007). 

23  GAO,  Tactical  Aircraft:  Opportunity  to  Reduce  Risks  in  the  Joint  Strike  Fighter 
Program  with  Different  Acquisition  Strategy ,  GAO-05-271  (Washington,  D.C.:  Mar.  15, 
2005). 
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that  these  programs  regularly  made  overly  optimistic  cost-estimating 
assumptions,  including  assumptions  about  their  ability  to  define 
requirements,  mature  technologies,  and  secure  the  funding  and  other 
resources  needed  to  develop  the  system  within  a  specified  l  ime  frame.24 
Similarly,  we  recently  testified  that  the  Navy  tends  to  underestimate  the 
costs  needed  to  construct  ships,  resulting  in  unrealistic  budgets  and  large 
cost  increases  after  ship  construction  has  begun. 26  We  noted  that  for  two 
major  ship  programs,  the  Navy  assumed  significant  savings  based  on 
efficiencies  that  did  not  materialize.  We  linked  these  optimistic 
assumptions  to  cost  growth  and  schedule  delays  in  a  number  of  these 
programs. 

Cost  estimates  that  lack  knowledge  and  rely  heavily  on  assumptions  have 
inherently  high  levels  of  risk  and  uncertainty.  Conducting  quantitative  risk 
and  uncertainty  analysis  provides  a  way  to  assess  the  variability  in  an 
estimate.  Using  this  type  of  analysis,  cost  estimators  can  model  such 
effects  as  a  schedule  slipping  or  a  key  technology  failing  to  materialize, 
thereby  identifying  a  range  of  likely  costs  around  an  estimate.  Presenting 
decision  makers  with  the  range  of  likely  costs  around  an  estimate  provides 
insight  into  the  amount  of  cost,  schedule,  and  technical  risks  they  are 
being  asked  to  accept  and  conveys  a  level  of  confidence  associated  with 
achieving  the  proposed  estimate.  A  range  of  costs  also  provides  a  basis  for 
deciding  how  much  funding  a  program  needs  to  be  successful. 

In  developing  cost  estimates,  the  services  and  the  CAIG  often  do  not 
present  decision  makers  with  the  range  of  costs  around  an  estimate.26 
Instead,  they  present  a  single,  or  point,  estimate  as  the  most  probable  cost, 
which  OSD  expects  to  be  a  50  percent  chance  that  actual  program  costs 
will  be  at  or  below  the  estimated  value.  Several  recent  studies  have 
questioned  DOD’s  approach  and  recommended  establishing  estimates  with 
higher  levels  of  confidence  under  the  assumption  that  the  result  will  be 


24  GAO,  Space  Acquisitions:  DOD  Needs  to  Take  More  Action  to  Address  Unrealistic 
Initial  Cost  Estimates  for  Space  Systems,  GAO-07-96  (Washington,  D.C.:  Nov.  17,  2006). 

25  GAO,  Defense  Acquisitions:  Realistic  Business  Cases  Needed  to  Execute  Navy 
Shipbuilding  Programs,  GAO-07-943T  (Washington,  D.C.:  July  24,  2007). 

26  For  management  to  make  good  decisions  about  programs,  our  Cost  Assessment  Guide 
calls  for  a  quantitative  risk  and  uncertainty  analysis  to  assess  variability  in  an  estimate. 
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more  realistic  estimates.27  For  example,  the  Defense  Acquisition 
Performance  Assessment  Panel  recommended  adjusting  program  cost 
estimates  to  reflect  “high  confidence” — defined  as  an  80  percent  chance  of 
completing  development  at  or  below  the  estimated  amount.  Requiring  a 
higher  confidence  level  could  provide  a  better  basis  for  determining 
program  funding,  but  only  if  the  quality  of  the  cost  estimate  is  sound  and 
the  underlying  risk  and  uncertainty  associated  with  the  estimate  are 
accurately  captured.  If  the  quality  of  the  estimate  is  poor  to  begin  with, 
simply  applying  a  higher  confidence  level  to  the  estimate  will  not  make  it 
any  more  realistic. 


DOD’s  Failure  to  Balance 
Needs  with  Resources 
Promotes  Unhealthy 
Competition  for  Funding 


In  prior  years,  we  have  reported  that  DOD  commits  to  more  programs  than 
its  resources  can  support.28  DOD’s  failure  to  balance  requirements  with 
available  resources  promotes  unhealthy  competition  among  programs  for 
funding.  Ultimately,  programs  tend  to  push  the  need  for  funding  to  the 
future  rather  than  limit  program  length  or  adjust  requirements. 


DOD’s  portfolio  of  weapon  system  programs  has  grown  over  the  past 
several  years  at  a  pace  that  far  exceeds  available  resources.  From  1992  to 
2007,  the  estimated  acquisition  costs  remaining  for  major  weapons 
programs  increased  almost  120  percent,  while  the  annual  funding  provided 
for  these  programs  only  increased  57  percent,  creating  a  fiscal  bow  wave 
that  may  be  unsustainable  (see  fig.  3).  If  this  trend  goes  unchecked  and 
fiscal  pressures  to  reduce  spending  continue  to  grow  as  expected, 
Congress  will  be  faced  with  a  difficult  choice  to  either  pull  funds  from 
other  federal  programs  to  support  DOD’s  acquisitions  or  accept  less 
warfighting  capability  than  promised. 


2‘  A  Defense  Science  Board  report  recommended  funding  space  acquisition  programs  at  the 
80  percent  confidence  level  ( Report  of  the  Defense  Science  Board  /Air  Force  Advisory 
Board,  Joint  Task  Force  on  Acquisition  of  National  Security  Space  Programs,  May  2003); 
and  subsequently  the  Air  Force  issued  guidance  in  March  2007  stating  that  an  80  percent 
confidence  level  is  the  objective  for  space  programs,  but  it  is  too  soon  to  assess  whether 
this  policy  has  resulted  in  more  realistic  estimates. 

28  GAO,  Future  Years  Defense  Program:  Risks  in  Operation  and  Maintenance  and 
Procurement  Programs,  GAO-Ol-33  (Washington,  D.C.:  Oct.  5,  2000),  and  Best  Practices: 
An  Integrated  Portfolio  Management  Approach  to  Weapon  System  Investments  Could 
Improve  DOD’s  Acquisition  Outcomes,  GAO-07-388  (Washington,  D.C.:  Mar.  30,  2007). 
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Figure  3:  Costs  Remaining  versus  Annual  Appropriations  for  Major  Defense  Acquisitions 


Dollars  (in  billions) 


Fiscal  year 

Costs  Remaining  for  Major  Defense  Acquisitions 
Annual  RDTE  and  Procurement  Appropriations 
Source:  DOD  (data);  GAO  (analysis  and  presentation). 


DOD’s  ability  to  prioritize  needs  within  available  resources  is  hampered 
because  its  processes  and  management  structures  for  determining 
capability  needs  and  allocating  resources  are  not  effectively  integrated. 
Capability  needs  are  formally  identified  and  validated  through  JCIDS, 
which  does  not  account  for  the  resources  that  will  be  needed  to  meet 
those  needs.  Instead,  resources  are  assessed  and  allocated  through  the 
PPBE  process,  which  as  we  have  reported  in  the  past,  is  service-centric 
and  does  not  effectively  link  resources  to  capabilities.  In  addition,  the 
PPBE  and  JCIDS  processes  are  led  by  different  organizations  within  DOD, 
as  is  the  process  for  executing  acquisitions,  making  it  difficult  to  hold  any 
one  person  or  organization  accountable  for  saying  no  to  a  proposed 
program  or  for  program  outcomes. 

DOD’s  failure  to  balance  its  needs  with  available  resources  promotes 
unhealthy  competition  among  and  within  the  services  to  get  funding  for 
new  programs  and  to  sustain  funding  for  existing  ones.  This  competition 
for  funding  creates  strong  incentives  for  service  and  program  officials  to 
establish  requirements  that  make  their  particular  weapon  systems  stand 
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out  from  others,  with  less  consideration  given  to  the  resources  that  will  be 
needed  to  develop  them.  Because  DOD’s  overall  funding  must  fit  within 
fiscal  constraints  of  the  FYDP  that  are  established  by  OMB,  services  place 
a  high  priority  on  the  appearance  of  affordability.  To  maintain  a 
competitive  edge,  services  develop  cost  estimates  that  will  fit  within 
established  funding  levels.  Also,  because  of  the  length  of  the  PPBE 
process,  initial  funding  for  a  program  is  often  established  in  the  FYDP 
before  the  program’s  initial  cost  estimated  is  developed  and  approved.  In 
addition,  program  officials  are  motivated  to  initiate  a  weapon  system 
development  program  because  once  initiated  a  program  is  in  a  more 
competitive  position  to  attract  high  levels  of  funding  and  management 
support.  Many  of  the  weapon  system  program  managers  who  responded  to 
a  GAO  survey  in  2005  cited  this  competitive  funding  environment  as  a  key 
obstacle  to  their  ability  to  effectively  manage  their  programs.29  For 
example,  program  managers  provided  comments  such  as  the  following  on 
competition  for  funding: 

•  “OSD  has  reduced  funding  without  any  understanding  or  appreciation 
for  program  impacts.  Funding  cuts  appear  arbitrary.” 

•  “OSD’s  near-term  execution  year  focus  results  in  great  instability.  In 
reality,  it  should  provide  more  strategic  vectors  for  the  department 
instead  of  short-term  adjustments  to  fix  more  tactical-level  funding 
needs.” 

•  “The  service  and  OSD  typically  cut  programs  to  pay  top  down  bills.” 

•  “There  is  no  such  thing  as  funding  stability  in  DOD.  Funding  reductions 
and  program  stretch-outs  are  the  norm  due  to  top  down  fiscal  bills  that 
occur  during  the  execution  year.” 

•  “Unstable  funding  results  in  pressure  to  do  aggressive  things  in  order  to 
minimize  the  impact  of  budget  cuts  on  schedule  and  performance.  I 
believe  this  has  been  a  major  factor  in  recent . . .  program  execution 
problems.” 

•  “When  funding  gets  tight,  we  have  been  considered  a  bill  payer  for 
others,  even  if  it  has  “broken”  our  program.” 

OSD  reviews  program  budgets  and  makes  adjustments  toward  the  end  of 
the  PPBE  cycle — often  only  2  or  3  months  before  the  budget  is  submitted 
to  Congress.  For  example,  in  December  2004,  OSD  cut  $30  billion  from  the 
2006  FYDP  for  many  of  its  major  acquisition  programs,  including 
prominent  aircraft  programs  like  the  F-22A  Raptor,  the  V-22  Osprey,  and 


29  GAO,  Best  Practices:  Better  Support  of  Weapon  System  Program  Managers  Needed  to 
Improve  Outcomes,  GAO-06-110  (Washington,  D.C.:  Nov.  30,  2005). 
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the  C-130J  Hercules.  Attempting  to  balance  investments  this  late  in  the 
process  often  leads  to  additional  churn  in  programs  and  encumbers  efforts 
to  meet  strategic  objectives  and  joint  needs. 

Instead  of  rethinking  performance  requirements  and  making  needed  trade¬ 
offs  within  and  among  programs,  DOD  and  the  services  often  pursue 
ambitious  solutions  and  technologies  that  result  in  cost  growth  and  the 
need  for  unplanned  funding.  Ultimately,  DOD  often  defers  costs  into  the 
future — beyond  the  FDYP — expecting  that  additional  funding  will  become 
available  when  it  is  needed.  According  to  DOD  and  service  officials,  what 
often  ends  up  happening  is  that  programs  have  to  extend  their  schedules 
to  spread  costs  out  over  time  or  reduce  requirements  to  hold  costs  down. 
These  strategies  ultimately  result  in  delaying  the  delivery  of  programs  or 
providing  the  warfighter  with  less  capability  than  promised. 


Proven  Practices  Help 
Ensure  Accurate  Cost 
Estimates  and 
Adequate  Program 
Funding 


Successful  commercial  companies  that  we  have  previously  reviewed 
achieved  adequate  and  stable  funding  for  product  development  programs 
by  following  practices  that  we  have  recognized  as  best  practices  for 
estimating  and  managing  program  costs.30  In  contrast  to  DOD,  these 
companies  followed  a  disciplined,  knowledge-based  approach  to 
estimating  program  costs.  Prior  to  initiating  product  development,  these 
companies  expected  cost  estimates  to  be  refined  based  on  increasing 
product  knowledge  and  assessed  the  estimates  at  multiple  gated  reviews. 
At  the  same  time,  the  companies  expected  proposed  programs  to  have  a 
manageable  development  cycle,  which  increases  the  predictability  of 
funding  needs  and  the  likelihood  of  program  success.  Understanding  that 
resources  are  limited,  the  companies  used  portfolio  management  practices 
to  make  tough  decisions  about  which  programs  to  pursue.  Once  programs 
were  approved,  these  companies  firmly  committed  to  fully  fund  them — 
which  they  often  achieved  by  committing  resources  incrementally  to 
successive  phases  of  a  program.  By  making  these  incremental 
commitments,  these  companies  were  able  to  limit  the  disruptions  that  can 
be  caused  by  poorly  performing  programs,  and  thus  maintained  a  high 
degree  of  stability  within  and  among  other  programs. 


30  See  Related  GAO  Products  at  the  end  of  this  report. 
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A  Disciplined,  Knowledge- 
Based  Approach  and 
Manageable  Development 
Cycles  Are  Key  to  Realistic 
Cost  Estimates 


As  part  of  our  best  practices  work  on  successful  product  development,  we 
have  found  that  following  a  disciplined,  knowledge-based  approach  that 
reduces  risk  and  uncertainty  over  time  is  integral  to  developing  realistic 
cost  estimates.  The  successful  companies  we  have  reviewed  conducted 
multiple  management  reviews  prior  to  initiating  product  development  to 
assess  the  business  case  of  each  proposed  product,  including  its  cost 
estimate.  They  typically  expected  cost  estimates  to  be  developed  early  on 
and  refined  over  time  as  knowledge  is  gained.  In  addition,  they  expected 
estimates  to  be  transparent  because  it  allowed  them  to  make  informed 
decisions  about  the  risks  and  uncertainty  associated  with  proposed 
programs  and  whether  they  should  pursue  or  cancel  them.  While  these 
companies  acknowledged  that  early  in  a  program’s  life  cycle  less  is  known 
about  the  product’s  design  and  technologies,  they  still  expected  to  receive 
rough  cost  estimates  at  that  stage.  They  expected  the  estimates  to  be 
revised  and  more  precise  as  knowledge  was  gained.  Before  committing  to 
product  development,  they  expected  the  level  of  uncertainty  in  the 
estimates  to  be  low. 

Our  Cost  Assessment  Guide  similarly  emphasizes  the  need  to  refine  cost 
estimates  based  on  knowledge  gained  over  time  and  account  for  risk  and 
uncertainty  in  the  estimates.  Early  cost  estimates  are  more  uncertain 
because  less  is  known  about  requirements  and  the  opportunity  for  change 
is  greater.  As  more  knowledge  is  gained,  programs  can  retire  some  risk 
and  reduce  the  potential  for  unexpected  cost  and  schedule  growth.  The 
best  practice  is  not  to  commit  to  product  development  until  the  level  of 
knowledge  is  high  and  the  level  of  uncertainty  is  low.  However,  our  past 
work  has  found  that  DOD  typically  commits  to  starting  programs  with  low 
levels  of  knowledge  and  high  levels  of  uncertainty  (see  fig.  4). 
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Figure  4:  Notional  Comparison  of  Cost  Estimating  Uncertainty  and  Levels  of 
Knowledge  at  Program  Start 


High 


Source:  GAO. 


Despite  having  high  levels  of  uncertainty,  DOD  commits  to  development 
programs  based  on  point  estimates  that  are  expected  to  represent  most 
likely  costs.  The  Air  Force’s  Cost  Risk  and  Uncertainty  Analysis 
Handbook  notes  that  decision  makers  need  point  estimates  when 
preparing  and  managing  a  budget  because  programs  are  funded  and 
executed  using  discrete  dollars,  not  ranges  of  dollars.  However,  DOD’s 
estimates  often  prove  to  be  understated  by  as  much  as  30  to  40  percent.  To 
make  more  informed  investment  decisions,  cost  estimating  best  practices 
call  for  estimating  a  range  of  possible  costs  around  a  point  estimate  to 
provide  information  about  the  levels  of  uncertainty  and  confidence.  As 
proposed  programs  gain  more  knowledge  and  progress  through  the  phases 
leading  up  to  the  start  of  product  development,  these  ranges  should 
narrow  (see  fig.  5).  According  to  cost-estimating  experts  and 
representatives  of  one  of  the  commercial  companies  we  spoke  with,  it  is 
better  to  overestimate  than  underestimate  costs. 
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Figure  5:  Range  of  Possible  Costs  Narrows  as  Knowledge  Is  Gained 


Officials  in  Motorola’s  Government  and  Enterprise  Mobility  Solutions 
business  unit  have  established  a  rigorous  review  process,  within  which 
proposed  products  are  expected  to  provide  cost  estimates  that  fall  within 
an  established  range  at  successive  review  gates.  At  the  first  review  gate, 
the  range  is  generous,  allowing  for  estimates  to  be  as  much  as  75  percent 
too  high  and  25  percent  too  low.  As  the  proposed  program  moves  through 
review  gates  and  more  becomes  known  about  requirements,  technologies, 
and  design,  the  established  range  narrows.  At  product  development 
initiation,  the  cost  estimate  is  expected  to  be  no  more  than  10  percent 
higher  and  5  percent  lower  than  what  actual  costs  will  be.  IBM  similarly 
allows  products  to  deviate  from  their  original  estimates  as  long  as  the 
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deviation  is  within  agreed-upon  limits,  which  are  established  in  a  contract 
between  senior  management  and  project  managers.31  Product  development 
teams  are  expected  to  execute  according  to  the  contract.  According  to 
IBM  officials,  program  cost  growth  of  more  than  5  or  10  percent  is 
generally  not  acceptable. 

Successful  companies  have  found  that  a  relatively  short,  manageable 
development  cycle  is  a  hallmark  of  an  executable  program.  Officials  at 
Motorola  told  us  that  cycle  time  is  one  of  the  key  metrics  they  use  when 
making  decisions  about  what  programs  to  pursue.  They  noted  that 
development  programs  longer  than  2  or  3  years  are  not  likely  to  be 
initiated  because  the  increased  uncertainty  would  make  it  too  difficult  to 
accurately  plan  for  and  execute  the  programs.  This  is  consistent  with  what 
we  found  at  other  successful  commercial  companies  as  well.  By 
constraining  development  cycles,  it  is  easier  to  more  accurately  estimate 
costs,  and  with  more  accurate  cost  estimates  companies  are  able  to  more 
precisely  predict  the  future  funding  needs  and  effectively  allocate 
resources. 

In  1998,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  stated  that  the  department’s  objective  must  and  will  be  to 
achieve  acquisition  cycle  times  no  longer  than  5  to  7  years,  noting  that 
long  acquisition  cycle  times  for  major  defense  programs  lead  to  higher 
costs  and  diminished  military  effectiveness.  DOD’s  acquisition  policy, 
revised  in  2003,  suggests  that  system  development  should  be  limited  to  a 
manageable  time  frame — about  5  years.  An  assessment  of  DOD’s 
acquisition  system  commissioned  by  the  Deputy  Secretary  of  Defense  in 
2006  similarly  recommended  that  programs  should  be  time-constrained 
with  development  cycles  no  longer  than  6  years  from  Milestone  A  to  low- 
rate  initial  production.32  According  to  the  assessment,  a  time-constrained 
development  cycle  can  reduce  pressure  on  investment  accounts  and 
increase  funding  stability  for  all  programs.  With  development  cycles  of  6 
years  or  less,  programs  could  be  fully  funded  within  the  FYDP  time  frame. 
In  addition,  constrained  cycle  times  would  force  programs  to  conduct 


31  IBM’s  contracts  are  not  focused  solely  on  program  costs;  they  also  establish  goals, 
objectives,  and  allowable  deviations  for  other  program  measures,  such  as  schedule  and 
revenue. 

32  Assessment  Panel  of  the  Defense  Acquisition  Performance  Assessment  Project  for  the 
Deputy  Secretary  of  Defense,  Defense  Acquisition  Performance  Assessment  Report  (Jan. 
2006). 
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more  detailed  systems  engineering  analyses,  increasing  the  likelihood  that 
their  requirements  can  be  met  with  available  resources.  We  recently 
reported  that  while  there  are  isolated  examples  of  DOD  programs  with 
cycle  times  shorter  than  5  years,  the  majority  of  programs  were  initiated 
with  much  longer  cycle  times.  In  some  cases,  these  longer  cycle  times 
have  been  extended.33  For  example,  FCS  was  expected  to  be  a  7.5-year 
development  effort  but  is  now  a  12-year  program  and  will  likely  be 
extended  again.  Unconstrained  and  lengthy  cycle  times  promote  program 
funding  instability — especially  when  considering  DOD’s  tendency  to 
change  requirements  and  funding  as  well  as  frequent  changes  in 
leadership. 


A  Portfolio-Based 
Investment  Approach  with 
Incremental  Commitments 
Supports  Adequate  and 
Stable  Program  Funding 


To  ensure  that  resources  are  available  to  fully  support  their  programs, 
successful  commercial  companies  we  reviewed  used  an  integrated 
portfolio  management  approach  to  make  incremental  investment 
decisions.  In  making  these  decisions,  the  companies  looked  across  their 
entire  product  portfolio  within  the  context  of  available  resources  to 
ensure  that  they  were  pursuing  a  balanced  mix  of  products  that  can 
optimize  the  return  on  their  investment.  Once  a  decision  was  made  to 
pursue  a  development  program,  the  companies  committed  to  fully  funding 
the  first  phase  of  that  program,  and  made  similar  commitments  prior  to 
initiating  each  successive  phase.  Before  programs  were  approved  to  enter 
the  next  phase  of  product  development,  Eli  Lilly  and  IBM  required 
program  officials  and  management  to  sign  a  contract  that  represented  the 
program’s  promise  to  deliver  a  given  product  that  would  meet  the 
customers’  needs  within  established  cost  and  time  constraints.  These 
contracts  also  represented  the  ongoing  commitment  of  management  to 
provide  the  necessary  funding  and  other  resources  needed  to  ensure  that 
the  program  could  be  successfully  executed. 


By  making  incremental  commitments  companies  are  better  positioned  to 
effectively  manage  and  maintain  stability  in  programs.  For  companies  like 
Eli  Lilly — which  tends  to  have  development  cycles  of  10  to  15  years  on 
average — the  practice  of  making  short-term,  incremental  commitments 
mitigates  many  of  the  cost,  schedule,  and  performance  risks  inherent  in 
long  development  programs.  When  Eli  Lilly  decides  to  pursue  a  new  drug 
development  program,  management  and  the  project  team  enter  into  a 


33  GAO,  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs, 
GAO-08-467SP  (Washington,  D.C.:  Mar.  31,  2008). 
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contract  that  identifies  deliverables,  time  frames,  and  the  costs  to  get  to 
the  next  milestone.  The  contract  represents  management’s  commitment  to 
fund  the  entire  phase.  IBM  similarly  allocates  funding  in  increments.  For 
example,  at  the  initial  review  gate,  funding  is  allocated  to  the  product 
development  team  to  support  the  development  of  a  sound  business  plan, 
which  is  expected  to  be  presented  at  the  next  review  gate  before  any 
further  funding  commitment  is  made. 

Making  incremental  commitments  also  allows  companies  the  flexibility  to 
effectively  manage  their  portfolio.  As  they  assess  the  value  and  progress  of 
each  investment  at  multiple  points  throughout  product  development, 
companies  are  able  to  make  tough  decisions  to  defer  or  terminate 
programs  and  rebalance  their  portfolio.  Terminating  low-value  or  poor¬ 
performing  programs  is  important  to  the  successful  companies  we  spoke 
with,  because  each  dollar  spent  on  a  failing  program  is  one  less  dollar 
available  for  use  elsewhere  in  the  company.  As  a  result,  they  emphasize 
the  need  to  make  these  tough  decisions  early.  For  example,  at  Eli  Lilly 
projects  are  terminated  at  early  points  in  the  review  process  when  it  is 
determined  that  their  critical  success  factors  cannot  be  achieved.  Because 
Eli  Lilly’s  projects  typically  have  a  high  degree  of  technical  risk,  only  about 
1  percent  of  those  that  start  early  development  actually  make  it  to  the 
marketplace. 

We  have  found  that  successful  portfolio  management  requires  a  strong 
governance  structure  with  committed  leadership  that  empowers  portfolio 
managers  to  make  decisions  about  the  best  way  to  invest  resources  and 
holds  those  managers  accountable  for  the  outcomes  they  achieve.  Because 
portfolio  managers  are  on  the  front  line,  several  of  the  companies  we  have 
reviewed  empower  these  managers  to  make  product  investment  decisions 
and  then  hold  them  accountable  for  outcomes,  not  just  for  individual 
products  but  also  for  the  overall  performance  of  the  portfolio.  These 
organizations  underscore  the  importance  of  holding  individuals 
accountable,  aligning  performance  expectations  with  organizational  goals, 
and  cascading  those  expectations  down  to  lower  levels. 

In  contrast,  DOD  approves  proposed  programs  without  adequately 
considering  its  overall  portfolio  and  commits  to  programs  with  less 
knowledge  of  cost  and  feasibility.  Moreover,  DOD  lacks  the  accountability 
that  commercial  companies  emphasize  is  necessary  to  ensure  successful 
outcomes.  Consequently,  DOD  starts  more  programs  than  current  and 
likely  future  resources  can  support  and  has  less  assurance  that  its 
investment  decisions  address  the  right  mix  of  warfighting  needs. 
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Conclusions 


At  a  time  when  the  federal  budget  is  strained  by  spending  needs  for  a 
growing  number  of  national  priorities,  it  is  imperative  that  DOD  get  the 
best  value  for  every  dollar  of  its  significant  investments.  Yet  DOD  has 
more  major  weapons  system  programs  in  its  portfolio  than  it  can  afford. 
All  too  often,  these  programs  incur  cost  increases  well  beyond  original 
funding  levels  and  significant  delays  in  delivering  weapon  systems  to  the 
warfighter.  These  outcomes  are  the  direct  result  of  DOD’s  failure  to 
prioritize  its  needs  within  resource  constraints  and  a  funding  process  that 
allows  programs  to  go  forward  with  unpredictable  cost  estimates  and 
lengthy  development  cycles — not  a  sound  basis  for  allocating  resources 
and  ensuring  program  stability.  Successful  commercial  companies  we 
have  reviewed  recognize  the  importance  of  having  adequate  knowledge 
about  requirements  and  available  resources  before  initiating  a 
development  program.  These  companies  have  learned  that  disciplined 
processes  with  early  management  review  points,  knowledge-based  cost 
estimates,  manageable  product  development  cycles,  and  a  portfolio 
approach  to  funding  product  development  efforts  are  key  to  achieving 
program  success.  While  DOD  has  recognized  that  more  accurate  cost 
estimates  coupled  with  manageable  development  cycles  could  improve 
program  outcomes,  it  has  yet  to  take  action.  Until  DOD  revises  its  policies 
and  processes  to  address  funding  imbalances  and  reform  its  funding 
approach,  programs  will  continue  to  experience  instability  and  delayed 
delivery  of  needed  capabilities  to  the  warfighter. 


Recommendations  for 
Executive  Action 

•  Develop  and  implement  a  strategy  to  bring  the  department’s  current 
portfolio  into  balance  by  aligning  the  number  of  programs  and  the  cost 
and  schedule  of  those  programs  with  available  resources.  In  developing 
and  implementing  a  strategy,  the  department  should  determine  ways  to 
prioritize  needs  and  identify  whether  the  budget  and  the  FYDP  should 
be  increased  to  more  accurately  reflect  the  actual  costs  of  current 
programs  or  whether  the  portfolio  of  current  programs  should  be 
reduced  and  lower-priority  programs  terminated  to  match  available 
resources. 

•  Require  that  all  new  programs  have  manageable  development  cycles, 
realistic  cost  estimates,  and  have  planned  and  programmed  full  funding 
for  the  entire  development  cycle. 


To  better  ensure  adequate  funding  for  DOD’s  major  weapon  system 
acquisition  programs  and  to  increase  the  likelihood  of  achieving 
successful  outcomes,  we  recommend  that  the  Secretary  of  Defense  take 
the  following  three  actions: 
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•  Require  all  cost  estimates  submitted  for  funding  a  program  at  milestone 
decisions  to  be  reported  as  a  range  of  likely  costs  and  reflect  the 
associated  levels  of  risk  and  uncertainty.  At  Milestone  A,  require 
estimates  that  allow  for  a  wide  range  of  likely  costs.  At  Milestone  B, 
require  estimates  that,  based  on  knowledge  gained,  are  more  precise — 
in  line  with  best  practice  standards. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  DOD  partially  concurred 
with  our  first  and  second  recommendations  and  non-concurred  with  the 
third.  DOD’s  partial  concurrences  are  rooted  in  the  belief  that  its  current 
policies  and  initiatives  address  our  recommendations.  We  agree  that 
aspects  of  DOD’s  current  policies  appear  consistent  with  our 
recommendations,  and  the  initiatives  could  contribute  to  better  program 
outcomes  if  implemented  appropriately.  However,  we  have  found  no 
evidence  of  widespread  adoption  of  these  policies  or  any  other  process 
that  would  better  ensure  adequate  funding  for  DOD’s  major  weapon 
system  acquisition  programs  and  increase  the  likelihood  of  achieving 
successful  outcomes. 


DOD  partially  concurred  with  our  first  recommendation — that  the 
Secretary  of  Defense  develop  and  implement  a  strategy  to  bring  the 
department’s  current  portfolio  into  balance  by  aligning  the  number  of 
programs  and  the  cost  and  schedule  of  those  programs  with  available 
resources.  In  its  written  response,  DOD  identified  the  2003  restructuring  of 
the  PPBE  process  as  one  initiative  it  has  taken  to  address  the  mismatch 
between  program  commitments  and  available  resources.  Yet,  since  the 
restructuring,  more  than  5  years — and  several  budget  cycles — have  lapsed, 
and  the  department  is  still  committed  to  more  programs  than  it  can 
support.  While  DOD  notes  that  seeing  the  effects  of  process  changes  will 
take  years,  the  department  does  not  indicate  how  many  additional  years 
are  needed  to  see  positive  results  or  what  incremental  changes  it  should 
be  held  accountable  for.  In  the  meantime,  the  department  continues  to  risk 
tax  dollars  and  delaying  the  delivery  of  needed  capabilities  to  the 
warfighter.  The  department  also  cites  several  other,  more  recent  initiatives 
like  Capability  Portfolio  Management,  Capital  Accounts,  and  Configuration 
Steering  Boards  that  are  intended  to  balance  needs  with  resources;  drive 
more  realistic,  cost-effective  plans  and  budgets;  limit  requirements  growth; 
and  improve  oversight.  While  we  believe  that  these  initiatives,  like  many 
before  them,  are  well  intentioned,  DOD  has  not  established  indicators  to 
measure  their  success,  and  it  is  unclear  how  the  initiatives  will  help  bring 
DOD’s  current  portfolio  into  balance.  Further,  we  are  concerned  that  they 
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do  not  go  far  enough  to  address  the  systemic  cultural  and  structural 
problems  identified  in  this  report. 

DOD  states  that  “external  influences”  can  cause  turbulence  in  program 
execution.  Specifically,  the  department  notes  that  it  does  not  control  the 
amount  of  Total  Obligation  Authority  (TOA)  available  in  the  FYDP  or  how 
much  funding  is  appropriated  by  Congress.  While  we  acknowledge  that 
decisions  made  by  OMB  and  Congress  can  directly  affect  program  funding, 
we  believe  that  our  recommendation  is  sound  regardless  of  the  total 
amount  of  funding  provided  to  the  department.  As  we  have  recently 
reported,  substantially  more  funding  has  been  committed  to  develop  new 
weapon  systems  since  2000,  yet  cost  overruns  and  schedule  delays  have 
increased  even  more.  By  focusing  on  external  influences  DOD  misses  the 
main  point  of  our  recommendation — to  align  the  number  of  programs  in 
the  current  portfolio  and  the  cost  and  schedule  of  those  programs  with 
available  resources. 

DOD  also  partially  concurred  with  our  second  recommendation — to 
require  all  new  programs  to  have  manageable  development  cycles,  realistic 
cost  estimates,  and  full  funding  for  the  entire  development  cycle.  Again, 
the  department’s  partial  concurrence  is  based  on  its  belief  that  certain 
DOD  policies  and  initiatives  respond  to  this  recommendation.  We  agree 
that  some  DOD  policies  and  initiatives  emphasize  a  knowledge-based 
approach  to  acquiring  weapon  systems,  but  acquisition  officials  do  not 
effectively  implement  these  policies.  DOD  programs  are  often  initiated 
with  development  cycle  times  that  are  much  longer  than  the  5  years  the 
department’s  policy  suggests.  As  we  note  in  this  report,  programs  with 
development  cycles  of  6  years  or  less  could  be  fully  funded  in  a  single 
FYDP  and  would  be  less  likely  to  experience  funding  instability.  DOD 
further  commented  that  the  current  acquisition  policy  requires  cost 
estimates  and,  where  required,  independent  cost  estimates  to  be 
completed  at  key  program  milestones  and  full  funding  to  be  programmed 
prior  to  Milestone  B.  However,  we  found  that  because  DOD’s  cost 
estimates  at  key  milestones  are  based  on  limited  knowledge  and  optimistic 
assumptions  about  requirements  and  technology,  they  often  significantly 
underestimate  true  costs  and  do  not  provide  the  necessary  foundation  for 
making  accurate  funding  commitments.  In  addition,  our  analysis  of  DOD’s 
selected  acquisition  reports  and  budget  data  found  that  DOD  often  does 
not  commit  full  funding  to  its  major  weapon  system  acquisitions  when 
they  are  initiated,  despite  the  department’s  policy  to  do  so. 
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While  the  department  generally  agreed  that  product  development  cycles 
should  be  “manageable,”  it  questioned  the  comparability  of  DOD  to  the 
private  sector  firms  we  reviewed  with  regard  to  industry  goals,  incentives, 
products,  and  services.  This  misses  the  point.  We  are  recommending  that 
development  cycles  for  all  new  DOD  programs  be  manageable — a  guiding 
principle  that  the  successful  commercial  companies  we  have  reviewed 
follow  to  enable  better  program  outcomes.  The  department’s  own 
acquisition  policy  suggests  development  cycles  of  about  5  years — an 
objective  the  department  established  a  decade  ago.  Similarly,  a  recent 
study  commissioned  by  DOD  recommended  that  programs  should  be  time- 
constrained  with  development  cycles  no  longer  than  6  years.  In  general, 
DOD  indicated  an  interest  in  obtaining  more  insight  into  our  methodology. 
Much  of  this  information,  including  information  about  the  companies  and 
commodities  that  we  have  reviewed  in  the  past,  can  be  found  in  most  of 
the  works  cited  in  the  related  GAO  products  section  of  this  report. 

DOD  did  not  concur  with  our  third  recommendation.  While  DOD  agreed 
that  risks  should  be  considered  at  key  milestones,  it  did  not  agree  that  cost 
estimates  submitted  at  milestone  decisions  should  be  reported  as  a  range 
of  likely  costs — a  wide  range  at  Milestone  A  based  on  limited  knowledge, 
narrowing  to  a  more  precise  range  at  Milestone  B  as  more  knowledge  is 
gained.  The  CAIG  accounts  for  cost  risk  by  examining  program  schedule 
durations,  technical  risks,  contract  vehicles,  incentives,  and  management 
structures.  However,  DOD’s  current  practice  of  presenting  a  single  point 
estimate  as  the  “most  likely  cost”  provides  decision  makers  with  limited 
insight  into  these  risks.  DOD  also  cited  concerns  that  a  certain  method  for 
calculating  cost  ranges  can  produce  misleading  results.  While  we  do  not 
advocate  a  specific  method  for  calculating  a  range  of  potential  costs,  we 
maintain  that  presenting  such  a  range  would  provide  decision  makers  with 
additional  knowledge  about  the  level  of  risk  in  a  proposed  program. 

Finally,  DOD  states  that  our  report  does  not  consider  certain  causal 
factors  bearing  on  the  estimates  we  cite  in  table  1.  It  is  unclear  how  DOD 
came  to  this  conclusion.  Our  report  states  that  inaccurate  cost  estimates 
are  often  the  result  of  limited  knowledge  about  requirements  and 
technologies.  We  also  note  that  conducting  early  disciplined  analyses, 
such  as  systems  engineering,  builds  knowledge  that  enables  a  developer  to 
identify  and  resolve  gaps  between  requirements  and  available  resources 
before  beginning  product  development.  As  knowledge  increases, 
uncertainty  and  associated  risks  in  the  cost  estimate  decrease.  Therefore, 
we  believe  that  following  a  disciplined  knowledge-based  process — a 
recommendation  we  have  made  repeatedly  in  our  reviews  of  DOD’s  major 
weapon  system  programs — would  obviate  most  of  the  causal  factors  DOD 
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cites,  including  “incomplete,  error-full,  or  volatile  requirements  or  program 
specification”  and  “engineering  change  proposals.” 

DOD’s  written  comments  are  reprinted  in  appendix  II.  The  Department 
also  provided  technical  comments,  which  were  incorporated  as 
appropriate. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense;  the 
Secretaries  of  the  Air  Force,  Army,  and  Navy;  and  the  Director  of  the 
Office  of  Management  and  Budget.  We  will  provide  copies  to  others  on 
request.  This  report  will  also  be  available  at  no  charge  on  GAO’s  Web  site 
at  http://www.gao.gov. 

If  you  have  any  questions  about  this  report  or  need  additional  information, 
please  contact  me  at  (202)  512-4841  or  sullivanm@gao.gov.  Key 
contributors  to  this  report  were  John  Oppenheim,  Assistant  Director; 
Travis  Masters;  Keith  Hudson;  Victoria  Klepacz;  Karen  Sloan;  Karen 
Richey;  and  John  Krump. 


Michael  J.  Sullivan 
Director,  Acquisition 
and  Sourcing  Management 
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This  report  assesses  the  extent  to  which  the  Department  of  Defense’s 
(DOD)  resource  allocation  approach  supports  stability  within  and  across 
its  major  weapon  system  acquisition  programs.  Specifically,  our  objectives 
were  to  (1)  assess  how  DOD  budgets  for  and  funds  its  major  weapon 
system  acquisition  programs,  (2)  identify  key  factors  that  influence  the 
effectiveness  of  this  approach,  and  (3)  identify  proven  processes  and 
practices  that  could  help  improve  DOD’s  ability  to  effectively  allocate 
resources  to  its  acquisition  programs. 

To  assess  the  effectiveness  of  DOD  process  for  funding  for  its  major 
weapon  system  acquisition  programs,  we  compared  initial  approved  cost 
estimates  for  20  of  the  95  major  weapons  programs  in  DOD’s  current 
portfolio  to  the  amount  of  development  funding  contained  in  each 
program’s  budget  documentation  from  the  year  in  which  development 
started — which  covers  the  initial  Future  Years  Defense  Program  (FYDP) 
time  frame.  The  20  programs  we  reviewed  initiated  system  development 
from  1999  through  2006  and  represent  almost  35  percent  of  the  total 
expected  cost  of  DOD’s  current  portfolio  of  major  weapon  system 
programs.  Due  to  data  limitations,  we  were  not  able  to  assess  a  number  of 
programs,  including  several  major  ship  acquisitions,  in  our  cost  estimate 
and  funding  analysis.  While  our  findings  may  not  be  generalized  to  all  of 
DOD’s  acquisition  programs,  we  believe  that  capturing  more  than  one- 
third  of  DOD’s  planned  investment  is  significant  and  provides  insights  into 
DOD’s  resource  allocation  approach  for  its  major  weapon  system 
programs.  To  gain  further  insights  into  the  impact  of  DOD’s  funding 
process  on  individual  programs,  we  conducted  more  detailed  analysis  for  5 
high-profile  programs,  which  represent  the  nearly  75  percent  of  DOD’s 
planned  investment  in  the  20  programs:  Global  Hawk,  Joint  Strike  Fighter 
(JSF),  Future  Combat  System  (FCS),  Warfighter  Information  Network- 
Tactical  (WIN-T),  and  Multi-mission  Maritime  Aircraft  (MMA).1  We 
obtained  cost  and  baseline  data  from  DOD’s  selected  acquisition  reports 
(SAR)  submitted  through  December  2007,  FYDP  and  program  funding  data 
from  the  President’s  Budget  exhibits  through  fiscal  year  2008,  and  program 


1  Global  Hawk,  an  Air  Force  unmanned  aircraft  system,  is  intended  to  provide  intelligence, 
surveillance,  and  reconnaissance  capabilities.  JSF  is  a  joint  Air  Force,  Navy,  and  Marine 
Corps  program  to  develop  and  field  stealthy  fighter  aircraft  to  replace  DOD’s  aging  fighter 
and  attack  aircraft.  FCS  is  an  Army  program  intended  to  provide  advanced,  networked 
combat  and  sustainment  systems;  unmanned  ground  and  air  vehicles;  and  unattended 
sensors  and  munitions.  WIN-T  is  intended  to  provide  the  Army  with  a  high-speed,  high- 
capacity  communications  network.  MMA  is  a  Navy  program  intended  to  provide  persistent 
antisubmarine  and  antisurface  warfare  and  intelligence,  surveillance,  and  reconnaissance 
capabilities. 
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cost  estimates  from  the  Office  of  the  Secretary  of  Defense’s  (OSD)  Cost 
Analysis  Improvement  Group  (CAIG)  and  the  service  cost  analysis  centers 
developed  to  support  each  program’s  Milestone  B  review.  Our  analysis 
began  with  cost  and  funding  data  from  the  fiscal  year  that  each  program 
entered  system  development — received  Milestone  B  approval — and  ended 
with  the  data  reported  in  the  December  2007  SARs.  We  also  examined  and 
compared  individual  program  cost  estimates  from  the  CAIG,  service  cost 
analysis  centers,  and  program  offices  to  determine  how  they  differed  from 
current  program  costs. 

To  better  understand  how  DOD  funds  its  weapon  system  acquisitions  and 
to  identify  the  key  factors  that  influence  its  effectiveness,  we  reviewed 
DOD  Acquisition  Regulations,  DOD  Financial  Management  Regulations, 
Title  10  of  the  United  States  Code,  Planning,  Programming,  Budgeting,  and 
Execution  guidance  from  Management  Information  Directive-913,  as  well 
as  individual  military  service  budgeting  policies  and  guidance.  We  also  met 
with  knowledgeable  officials  from  the  OSD  CAIG;  OSD  Program  Analysis 
and  Evaluation;  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics;  the  military  service  cost  analysis 
centers;  military  service  planning  and  programming  organizations;  the 
Defense  Acquisition  University;  the  Brookings  Institution;  the  Institute  for 
Defense  Analysis;  and  the  Naval  Postgraduate  School. 

To  identify  proven  cost  estimating  and  budgeting  processes  and  practices 
that  could  be  used  by  DOD  to  improve  its  resource  allocation  process,  we 
relied  on  our  prior  work  in  best  practices  at  successful  commercial 
companies  and  reviewed  documentation  and  interviews  from  previous 
GAO  best  practices  work.  We  also  conducted  follow-up  interviews  with 
key  officials  from  Eli  Lilly,  IBM,  and  Motorola2  to  gain  a  better 
understanding  of  how  they  prioritize  projects  for  funding  and  ensure  that 
each  product  receives  and  maintains  adequate  funding.  We  also  relied  on 
our  Cost  Assessment  Guide — which  provides  a  cost-estimating 
methodology  based  on  best  practices — and  numerous  other  GAO 
products,  including  our  extensive  body  of  best  practices  work. 


2  Eli  Lilly  is  one  of  the  largest  corporations  in  the  world  and  engages  in  pharmaceutical 
research  and  development  around  the  world;  IBM  is  the  largest  supplier  of  hardware, 
software,  and  information  technology  services;  and  Motorola  is  a  Fortune  100  global 
communications  leader.  We  also  contacted  the  other  two  companies  that  provided  input  to 
our  prior  review — Proctor  &  Gamble  and  Caterpillar — but  relevant  officials  were  not 
available  to  meet  with  us  for  follow-up  discussions  during  the  time  frames  of  our  review. 


Page  35 


GAO-08-619  Defense  Acquisitions 


Appendix  I:  Objectives,  Scope,  and 
Methodology 


We  conducted  this  performance  audit  from  June  2007  to  May  2008  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3000 


ACQUISITION, 
TECHNOLOGY 
AND  LOGISTICS 


JJN 13  m 


Mr.  Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr.  Sullivan: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report  GAO- 
08-619,  “DEFENSE  ACQUISITIONS:  A  Knowledge-Based  Funding  Approach  Could 
Improve  Major  Weapon  System  Program  Outcomes,”  dated  May  15,  2008  (GAO  Code 
120665). 

The  Department  partially  concurs  with  two  recommendations  and  nonconcurs  with 
the  other  draft  report  recommendation.  The  rationale  for  DoD’s  position  is  included  in 
the  enclosure. 

The  Department  appreciates  the  opportunity  to  comment  on  the  draft  report. 
Technical  comments  were  provided  separately.  My  point  of  contact  for  this  effort  is 
Mr.  Joseph  A.  Alfano,  703-697-3343. 

Sincerely, 


Dr.  Nancy  L.  Spruill 
Director, 

Acquisition  Resources  &  Analysis 


Enclosure: 
As  stated 
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GAO  DRAFT  REPORT  DATED  MAY  15, 2008 
GAO-08-619  (GAO  CODE  120665) 


“DEFENSE  ACQUISITIONS:  A  KNOWLEDGE-BASED  FUNDING 
APPROACH  COULD  IMPROVE  MAJOR  WEAPON  SYSTEM  PROGRAM 
OUTCOMES” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 

RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense  develop  and 
implement  a  strategy  to  bring  the  department’s  current  portfolio  into  balance  by  aligning  the 
number  of  programs  and  the  cost  and  schedule  of  those  programs  with  available  resources.  In 
developing  and  implementing  a  strategy,  the  Department  should  determine  ways  to  prioritize 
needs  and  identify  whether  the  budget  and  the  Future  Years  Defense  Program  should  be 
increased  to  more  accurately  reflect  the  actual  costs  of  current  programs  or  whether  the  portfolio 
of  current  programs  should  be  reduced  and  lower  priority  programs  terminated  to  match 
available  resources,  (p.  28/GAO  Draft  Report) 

POD  RESPONSE:  Partially  concur.  The  Department  goes  to  great  lengths  in  order  to  develop 
overall  investment  strategy  and  guidance,  then  flows  that  guidance  to  the  appropriate  levels,  and 
then  prioritizes  requirements  in  order  to  “align  the  number  of  programs  and  the  cost  and  schedule 
of  those  programs  with  available  resources.”  The  restructure  of  the  Planning,  Programming,  and 
Budgeting  System  into  the  Planning,  Programming,  Budgeting,  and  Execution  process  was  one 
of  several  initiatives  the  Department  has  taken  towards  this  end;  however,  positive  results  from 
these  types  of  process  changes  will  take  years  and  experience  across  several  programs  in  order  to 
validate  their  positive  effects.  Yet  while  the  Department  has  control  over  its  programming  and 
budgeting  processes  (within  limits  set  by  the  Office  of  Management  and  Budget)  it  does  not 
control  the  amount  of  Total  Obligation  Authority  available  over  the  Future  Years  Defense 
Program,  nor  how  much  funding  is  appropriated  by  Congress.  At  times,  these  external 
influences  will  also  cause  turbulence  in  program  execution. 

Over  the  last  several  years  the  Department  has  taken  on  several  initiatives  to  address  this  GAO 
recommendation.  A  recent  Defense  initiative  known  as  Capability  Portfolio  Management  is 
intended  to  provide  an  enterprise-level,  horizontal  (cross-component)  view  of  the  Department  to 
better  balance  and  harmonize  the  Departments  capability  needs  with  existing  and  planned  force 
management  and  development  efforts  and  produce  strategically  aligned  outcomes  optimized  for 
the  enterprise.  As  of  February  2008,  Department  leadership  has  established  nine  Capability 
Portfolio  Managers  aligned  to  the  nine  Tier  1  Joint  Capability  Areas  (the  Department’s 
capabilities  language),  formalized  four  and  is  testing  the  other  five.  Capability  Portfolio 
Managers  are  focused  on  highlighting  opportunities  to  better  integrate,  coordinate  and 
synchronize  programs  to  strategic  intent  and  capability  priorities  within  resource  and  time 
constraints. 
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With  respect  to  budgeting,  more  realistic,  cost-effective  plans  and  budgets  are  goals  for  the 
integrated  Planning,  Programming,  Budgeting  and  Execution  process.  Establishing  Capital 
Accounts  will  result  in  more  stable,  predictable  acquisition  lifecycle  management  program 
execution,  and  the  process  would  benefit  from  this  guaranteed  funding  stream.  Pilot  programs 
provide  valuable  lessons  learned  to  apply  capital  accounts  to  a  wide  variety  of  acquisition  and 
sustainment  programs.  Current  initiatives  include  coordination  with  the  Office  of  Management 
and  Budget  Program  Assessment  Rating  Tool  and  establishment  of  an  authoritative  financial 
information  source  by  integrating  transactional-level  accounting  data.  Further,  Configuration 
Steering  Boards  have  been  implemented  across  all  major  programs  to  limit  cost  and  requirements 
growth  through  senior  level  evaluation  of  trade-offs  and  opportunities  to  descope  programs  and 
seek  savings. 

In  the  oversight  of  Major  Defense  Acquisition  Programs  and  the  Defense  Acquisition  Board 
process,  multiple  initiatives  are  being  implemented  to  improve  and  standardize  the  milestone 
decision  process.  The  Defense  Acquisition  Executive  Summaries  system  is  being  reengineered 
to  ensure  visibility  for  senior  leaders.  The  system  will  help  faoilitate  information  sharing, 
accurate  input  of  data,  and  tracking  of  negative  trends  to  focus  on  problem  areas.  Systems  to 
model  time  and  new  technology  factors  are  being  developed  to  budget  effectively  to  changing 
needs  in  a  rapidly  evolving  technological  development  environment. 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense  require  that  all 
new  programs  have  manageable  development  cycles,  realistic  cost  estimates,  and  have  planned 
and  programmed  full  funding  for  the  entire  development  cycle,  (p.  28/GAO  Draft  Report) 

POD  RESPONSE:  Partially  concur.  Current  DoD  acquisition  policy,  as  articulated  in  DoDI 
5000.2,  establishes  an  evolutionary,  event-based  acquisition  approach  as  the  Department’s 
preferred  strategy  for  rapid  acquisition  of  mature  technology  for  the  user.  Additionally,  current 
policy  requires  cost  estimates  and,  where  required,  independent  cost  estimates  to  be  completed  at 
key  program  milestones  and  full  funding  to  be  programmed  prior  to  Milestone  B . 

These  existing  policies  are  consistent  with  the  GAO’s  recommendation.  Additionally,  the 
Department  has  recently  introduced  a  number  of  new  policies  that  we  believe  will  substantively 
contribute  to  more  predictable  program  outcomes: 

-  Our  prototyping  and  competition  policy  requires  program  managers  to  plan  for  two  or  more 
competing  teams  to  produce  prototypes  of  key  system  elements  up  to  or  through  Milestone  B. 
The  intent  is  to  reduce  technical  risk,  validate  cost  estimates,  evaluate  manufacturing  processes, 
and  refine  requirements. 

-  We  have  established  Configuration  Steering  Boards  for  all  ACAT I  programs  to  review 
proposed  requirements  changes,  a  significant  contributor  to  increased  cost  and  extended 
development  cycles,  and  any  technical  configuration  changes  that  have  the  potential  to  result  in 
increases  to  cost  and  or  lengthier  schedules.  Such  changes  will  generally  be  rejected,  deferring 
them  to  future  development  increments. 

Finally,  the  Department  is  considering  policy  initiatives  that  are  designed  to  reduce  program 
instability  and  improve  schedule  and  cost  predictability.  These  initiatives  include:  (1)  moving 
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the  Preliminary  Design  Review  before  Milestone  B,  to  improve  our  understanding  of  cost  and 
technical  requirements  before  program  initiation;  and  (2)  requiring  a  Milestone  Decision 
Authority-conducted  post-Critical  Design  Review  Assessment  in  order  to  ensure  that  programs 
are  proceeding  through  the  System  Development  and  Demonstration  phase  consistent  with  sound 
engineering  practice  and  the  approved  acquisition  program  baseline. 

Together  we  believe  these  new  or  pending  policies  will  enhance  existing  policy,  and 
substantively  contribute  to  improved  cost  estimates,  better  informed  decision  making  and  shorter 
cycle  times. 

Recommendation  2  includes  a  provision  that  all  new  development  programs  have  “manageable 
development  cycles.”  This  conclusion  is  based  on  the  commercial  product  development  models 
reviewed  by  GAO  and  mentioned  in  the  report,  specifically  those  from  Eli  Lilly,  IBM,  and 
Motorola.  While  we  generally  agree  that  DoD  product  development  cycles  should  be 
“manageable,”  these  development  cycles  may  bear  little  relation  to  those  of  the  three  commercial 
firms  cited  by  GAO  in  the  report. 

The  use  by  GAO  of  three  commercial  firm  product  development  models  as  a  basis  for 
determining  ‘best  practice’  in  DoD  has  several  flaws.  First,  DoD  as  a  public  sector  institution,  in 
contrast  to  a  private  sector  film,  has  key  differences  in  goals  and  incentives.  All  of  the  DoD 
weapon  system  programs  cited  in  the  report  are  developed  through  contracts  from  the  public  to 
the  private  sector,  which  contain  significantly  different  incentives  and  provisions  than  the 
product  development  programs  typically  undertaken  internally  by  firms  in  the  private  sector.  It 
may  be  more  appropriate  that  instead  of  benchmarking  DoD  against  private  sector  firms,  GAO 
should  be  benchmarking  DoD  against  best  practices  in  the  public  sector  throughout  the  world. 

Further  the  commodity  classes  for  comparison  to  DoD  development  programs  for  major  weapon 
systems  that  GAO  selected,  we  believe,  are  not  a  good  comparison.  The  three  firms  cited  in  the 
report  are  in  the  pharmaceutical,  computer  products  and  services,  and  the  radio  and  cellular 
telephony  sectors.  Yet  the  implication  of  the  report  is  that  the  development  cycles  in  these 
product  sectors  can  be  directly  applied  to  systems  like  the  Joint  Strike  Fighter  program.  At  a 
minimum,  GAO  might  better  examine  commercial  product  developments  in  comparable 
commodity  sectors,  such  as  the  commercial  aircraft  sector,  to  avoid  drawing  conclusions  that 
may  be  invalid  based  on  analogies  to  dissimilar  commodity  sectors. 

In  order  to  more  thoroughly  review  the  study,  the  DoD  requested  that  the  GAO  share  its 
methodology,  raw  data,  interview  notes,  detailed  analyses,  working  papers,  etc.  Although  to  date 
the  GAO  has  not  provided  the  aforementioned  materials,  the  Department  would  be  pleased  to 
work  with  GAO  to  better  understand  the  details  of  such  a  comparison. 

RECOMMENDATION  3:  The  GAO  recommends  that  the  Secretary  of  Defense  require  all  cost 
estimates  submitted  for  funding  a  program  at  Milestone  decisions  to  be  reported  as  a  range  of 
likely  costs  and  the  associated  levels  of  risk  and  uncertainty.  At  Milestone  A,  require  estimates 
that  allow  for  a  wide  range  of  likely  costs.  At  Milestone  B,  require  estimates  that,  based  on 
knowledge  gained,  are  more  precise~in  line  with  best  practice  standards,  (p.  28/GAO  Draft 
Report) 
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Appendix  II:  Comments  from  the  Department 
of  Defense 


POD  RESPONSE:  Non-concur.  While  we  and  the  DoD  Cost  Analysis  Improvement  Group 
(CAIG)  agree  with  the  spirit  of  the  recommendation  to  consider  risks  at  key  milestones  in 
development  programs;  however  the  GAO  recommendation  implies  that  the  DoD  should 
calculate  a  range  of  costs  and  associated  confidence  levels  using  Monte  Carlo  analysis 
techniques.  The  CAIG  has  observed  limitations  to  this  analysis  methodology  when  used  in 
practice:  Most  mathematical  implementations  of  this  methodology  assume  no  correlation  of 
work  breakdown  structure  elements  in  developing  distributions  of  potential  cost  outcomes.  In 
most  cases  this  simplifying  assumption  is  incorrect,  particularly  for  cost  estimates  developed  for 
product-oriented  work  breakdown  structures,  as  are  commonly  used  in  the  DoD.  This 
assumption  results  in  calculated  distributions  of  potential  cost  outcomes  that  are  “too  narrow” 
relative  to  the  actual  distribution  of  cost  outcomes,  providing  misleading  results. 

The  CAIG  does  routinely  calculate  cost  risks  for  major  milestone  reviews  by  examining  the  key 
structural  elements  of  cost  estimates,  including  schedule  durations,  technical  risks,  contract 
vehicles,  incentives,  and  management  structures.  In  certain  situations,  the  CAIG  has  reported  a 
range  of  life-cycle  costs  because  of  the  poor  quality  of  the  definition  of  the  program  to  be 
undertaken  in  the  Cost  Analysis  Requirements  Description  document.  For  example,  the  CAIG 
reported  a  likely  range  of  Future  Combat  Systems  (FCS)  development  costs  on  several 
occasions,  a  fact  that  GAO  did  not  include  in  its  report.  In  summary,  the  CAIG  tailors  the  risk 
analysis  to  the  specific  program  and  milestone  at  hand,  calculating  and  presenting  ranges  of  costs 
when  it  is  appropriate  to  do  so.  The  specification  of  a  single  methodology  and  practice  to  be 
used  at  each  and  every  milestone  review  of  a  Major  Defense  Acquisition  Program  is  not 
appropriate. 

In  the  GAO  report.  Table  1,  Development  Costs  Estimates  and  Baselines  for  20  Major  Weapon 
System  Programs,  contains  information  on  development  cost  estimates  and  cost  baselines. 
Unfortunately,  the  report  does  not  consider  causal  factors  bearing  on  the  estimates:  (1) 
methodological  issues,  (2)  incomplete,  errorful,  or  volatile  requirements  or  program 
specification,  (3)  engineering  change  proposals,  (4)  Congressionally  mandated  changes  in 
program  funding  levels  or  unit  quantities,  etc.  Additionally,  the  report  does  not  specify  the 
metrics  whereby  a  cost  estimate  is  considered  as  acceptable. 

In  order  to  assist  in  maintaining  an  awareness  of  overall  risk  with  its  acquisition  programs,  the 
Air  Force  is  evaluating  the  idea  of  setting  a  goal  of  annual  updates  to  program  cost  estimates  for 
major  defense  acquisition  programs,  versus  the  DoD  5000  series  requirement  to  update  for  each 
milestone  review.  This  should  bring  newly-acquired  knowledge  to  bear  more  quickly  and  bring 
problems  to  light  in  a  more  timely  manner. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday,  GAO  posts 
newly  released  reports,  testimony,  and  correspondence  on  its  Web  site.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products  every  afternoon,  go 
to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each. 

A  check  or  money  order  should  be  made  out  to  the  Superintendent  of 
Documents.  GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or 
more  copies  mailed  to  a  single  address  are  discounted  25  percent.  Orders 
should  be  sent  to: 

U.S.  Government  Accountability  Office 

441  G  Street  NW,  Room  LM 

Washington,  DC  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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